COSC 426: Augmented Reality

Mark Billinghurst

mark.billinghurst@hitlabnz.org

August 16th 2013

Lecture 6: AR User Interface Guidelines

AR Authoring Tools   Low Level Software Libraries

  osgART, Studierstube, MXRToolKit   Plug-ins to existing software

  DART (Macromedia Director), mARx, Unity,   Stand Alone

  AMIRE, BuildAR, Metaio Creator etc   Rapid Prototyping Tools

  Flash, OpenFrameworks, Processing, Arduino, etc   Next Generation

  iaTAR (Tangible AR)

BuildAR

  http://www.buildar.co.nz/   Stand alone application   Visual interface for AR model viewing application   Enables non-programmers to build AR scenes

Scene Graph Example

osgART Approach: AR Scene Graph

Video Geode

Root

Transform

3D Object

Virtual Camera

Projection matrix from tracker calibration

Transformation matrix updated from marker tracking in realtime Video

Layer Full-screen quad with live texture updated from Video source

Orthographic projection

AR Interaction

  Designing AR System = Interface Design   Using different input and output technologies

  Objective is a high quality of user experience   Ease of use and learning   Performance and satisfaction

AR Information Browsers   Information is registered to

real-world context   Hand held AR displays

  Interaction   Manipulation of a window

into information space   Applications

  Context-aware information displays

Rekimoto, et al. 1997

3D AR Interfaces

  Virtual objects displayed in 3D physical space and manipulated   HMDs and 6DOF head-tracking   6DOF hand trackers for input

  Interaction   Viewpoint control   Traditional 3D user interface

interaction: manipulation, selection, etc.

Kiyokawa, et al. 2000

Augmented Surfaces

  Rekimoto, et al. 1998   Front projection  Marker-based tracking  Multiple projection surfaces

Tangible AR: Time-multiplexed Interaction

  Use of natural physical object manipulations to control virtual objects

  VOMAR Demo  Catalog book:

-  Turn over the page

  Paddle operation: -  Push, shake, incline, hit, scoop

  Space-multiplexed   Many devices each with one function

-  Quicker to use, more intuitive, clutter -  Real Toolbox

  Time-multiplexed   One device with many functions

-  Space efficient -  mouse

Object Based Interaction: MagicCup   Intuitive Virtual Object Manipulation

on a Table-Top Workspace

  Time multiplexed  Multiple Markers

-  Robust Tracking

  Tangible User Interface -  Intuitive Manipulation

  Stereo Display -  Good Presence

Our system

 Main table, Menu table, Cup interface

Wrap-up   Browsing Interfaces

  simple (conceptually!), unobtrusive

  3D AR Interfaces   expressive, creative, require attention

  Tangible Interfaces   Embedded into conventional environments

  Tangible AR  Combines TUI input + AR display

Designing AR Interfaces

The Interaction Design Process

How Would You Design This?

  Put nice AR Picture here – and video

Or This?

experiences

applications

tools

components

Sony CSL © 2004

Building Compelling AR Experiences

Tracking, Display

Authoring

Interaction

AR Interaction Design

  Designing AR System = Interface Design   Using different input and output technologies

  Objective is a high quality of user experience   Ease of use and learning   Performance and satisfaction

AR UI Design  Consider your user   Follow good HCI principles  Adapt HCI guidelines for AR  Design to device constraints  Using Design Patterns to Inform Design  Design for you interface metaphor  Design for evaluation

Consider Your User   Consider context of user

  Physical, social, emotional, cognitive, etc

  Mobile Phone AR User   Probably Mobile  One hand interaction   Short application use  Need to be able to multitask  Use in outdoor or indoor environment  Want to enhance interaction with real world

Good HCI Principles  Affordance  Reducing cognitive overload  Low physical effort  Learnability  User satisfaction  Flexibility in use  Responsiveness and feedback  Error tolerance

Norman’s Principles of Good Practice   Ensure a high degree of visibility

-  allow the user to work out the current state of the system and the range of actions possible.

  Provide feedback -  continuous, clear information about the results of actions.

  Present a good conceptual model -  allow the user to build up a picture of the way the system holds

together, the relationships between its different parts and how to move from one state to the next.

  Offer good mappings -  aim for clear, natural relationships between actions the user

performs and the results they achieve.

Adapting Existing Guidelines   Mobile Phone AR

  Phone HCI Guidelines  Mobile HCI Guidelines

  HMD Based AR   3D User Interface Guidelines   VR Interface Guidelines

  Desktop AR  Desktop UI Guidelines

iPhone Guidelines

  Make it obvious how to use your content.   Avoid clutter, unused blank space, and busy

backgrounds.   Minimize required user input.   Express essential information succinctly.   Provide a fingertip-sized target area for all links and

controls.   Avoid unnecessary interactivity.   Provide feedback when necessary

Applying Principles to Mobile AR   Clean   Large Video View   Large Icons   Text Overlay   Feedback

AR vs. Non AR Design

  Design Guidelines  Design for 3D graphics + Interaction  Consider elements of physical world   Support implicit interaction

Characteristics Non-AR Interfaces AR Interfaces

Object Graphics Mainly 2D Mainly 3D

Object Types Mainly virtual objects Both virtual and physical objects

Object behaviors Mainly passive objects Both passive and active objects

Communication Mainly simple Mainly complex

HCI methods Mainly explicit Both explicit and implicit

Maps vs. Junaio

  Google Maps   2D, mouse driven, text/image heavy, exocentric

  Junaio   3D, location driven, simple graphics, egocentric

Design to Device Constraints   Understand the platforms and design for limitations

 Hardware, software platforms   Eg Handheld AR game with visual tracking

  Use large screen icons   Consider screen reflectivity   Support one-hand interaction   Consider the natural viewing angle   Do not tire users out physically   Do not encourage fast actions   Keep at least one tracking surface in view

31

Art of Defense Game

Design Patterns “Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem in such a way that you can use this solution a million times over, without ever doing it the same way twice.”

– Christopher Alexander et al.

Use Design Patterns to Address Reoccurring Problems

C.A. Alexander, A Pattern Language, Oxford Univ. Press, New York, 1977.

Handheld AR Design Patterns Title Meaning Embodied Skills Device Metaphors Using metaphor to suggest available

player actions Body A&S Naïve physics

Control Mapping Intuitive mapping between physical and digital objects

Body A&S Naïve physics

Seamful Design Making sense of and integrating the technological seams through game design

Body A&S

World Consistency

Whether the laws and rules in physical world hold in digital world

Naïve physics Environmental A&S

Landmarks Reinforcing the connection between digital-physical space through landmarks

Environmental A&S

Personal Presence The way that a player is represented in the game decides how much they feel like living in the digital game world

Environmental A&S Naïve physics

Living Creatures Game characters that are responsive to physical, social events that mimic behaviours of living beings

Social A&S Body A&S

Body constraints Movement of one’s body position constrains another player’s action

Body A&S Social A&S

Example: Seamless Design

  Design to reduce seams in the user experience   Eg: AR tracking failure, change in interaction mode

  Paparazzi Game   Change between AR tracking to accelerometer input

Example: Living Creatures

  Virtual creatures should respond to real world events   eg. Player motion, wind, light, etc   Creates illusion creatures are alive in the real world

  Sony EyePet   Responds to player blowing on creature

35

Designing for Children  Development Psychology Factors

•  Motor Abilities •  Spatial Abilities •  Logic Abilities •  Attention Abilities

Radu, Iulian, and Blair MacIntyre. "Using children's developmental psychology to guide augmented-reality design and usability." Mixed and Augmented Reality (ISMAR), 2012 IEEE International Symposium on. IEEE, 2012.

Motor Abilities Skill Type Challenging AR Interaction Multiple hand coordination Holding phone in one hand and

using another hand to move marker

Hand-eye coordination Using a marker to intercept a moving object

Fine motor skills Moving a marker on a specified path

Gross motor skills and endurance Turning body around to look at a panorama

Spatial Abilities Skill Type Challenging AR Interaction Spatial memory Remembering the configuration of a large

virtual space while handheld screen shows a limited view

Spatial Perception Understanding when a virtual item is on top of a physical item

Spatial Visualization Predict when virtual objects are visible by other people or virtual characters

Attention and Logic

Skill Type Challenging AR Interaction Divided attention Playing an AR game, and making sure to

keep marker in view so tracking is not lost

Selective and executive attention

Playing an AR game while moving outdoors

Skill Type Challenging AR Interaction Remembering and reversing Remembering how to recover from tracking

loss

Abstract over concrete thinking

Understanding that virtual objects are computer generated, and they do not need to obey physical laws

Attention Abilities

Logic and Memory

Design Considerations  Combining Real and Virtual Images

•  Perceptual issues  Interactive in Real-Time

•  Interaction issues  Registered in 3D

•  Technology issues

AR Perceptual Issues

AR and Perception  Creating the illusion that virtual images are

seamlessly part of the real world •  Must match real and virtual cues

•  Depth, occlusion, lighting, shadows..

AR as Perception Problem   Goal of AR to fool human senses – create

illusion that real and virtual are merged   Depth

  Size  Occlusion   Shadows   Relative motion   Etc..

Perceptual Issues

 Combining multiple display modes •  Direct View, Stereo Video View, Graphics View

 Conflict between display modes •  Mismatch between depth cues

Perceptual Issues  Static and Dynamic registration mismatch  Restricted Field of View  Mismatch of Resolution and Image clarity  Luminance mismatch  Contrast mismatch  Size and distance mismatch  Limited depth resolution  Vertical alignment mismatches  Viewpoint dependency mismatch

Depth Cues  Pictorial: visual cues

•  Occlusion, texture, relative brightness  Kinetic: motion cues

•  Relative motion parallax, motion perspective  Physiological: motion cues

•  Convergence, accommodation  Binocular disparity

•  Two different eye images

Use the Following Depth Cues   Movement parallax.   Icon/Object size (for close objects)   Linear perspective

  To add side perspective bar.

  Overlapping  Works if the objects are big enough

  Shades and shadows.  Depends on the available computation

Provide Perspective Cue

  Eg ground plane grid

Depth Perception

Information Presentation •  Amount of information

•  Clutter, complexity • Representation of information

•  Navigation cues, POI representation •  Placement of information

•  Head, body, world stabilized •  View combination

•  Multiple views

Twitter 360

  www.twitter-360.com   iPhone application   See geo-located tweets in real world   Twitter.com supports geo tagging

Wikitude – www.mobilizy.com

Blah

Blah

Blah Blah Blah

Blah Blah

Blah

Blah Blah Blah

Blah Blah

Blah

Blah Blah

Blah

Blah Blah

Blah

Blah Blah

Information Filtering

Information Filtering

Outdoor AR: Limited FOV

Possible solutions   Overview + Detail   spatial separation; two views

  Focus + Context  merges both views into one view

  Zooming   temporal separation

  TU Graz – HIT Lab NZ - collaboration   Zooming panorama   Zooming Map

Zooming Views

AR Interaction Issues

  Interface Components  Physical components  Display elements

- Visual/audio   Interaction metaphors

Physical Elements

Display Elements Interaction

Metaphor Input Output

AR Design Elements

Physical Elements

AR Design Space

Reality Virtual Reality

Augmented Reality

Physical Design Virtual Design

Design of Objects   Objects

  Purposely built – affordances   “Found” – repurposed   Existing – already at use in marketplace

  Affordance   The quality of an object allowing an action-

relationship with an actor   An attribute of an object that allows people to

know how to use it -  e.g. a door handle affords pulling

Norman on Affordances "...the term affordance refers to the perceived

and actual properties of the thing, primarily those fundamental properties that determine just how the thing could possibly be used. [...] Affordances provide strong clues to the operations of things. Plates are for pushing. Knobs are for turning. Slots are for inserting things into. Balls are for throwing .. " (Norman, The Psychology of Everyday Things 1988, p.9)

Physical vs. Virtual Affordances   Physical affordances

-  Physical and material aspects of real object  Virtual affordance

-  Visual and perceived aspects of digital objects

 AR is mixture of physical and virtual affordances   Physical

-  Tangible controllers and objects   Virtual

-  Virtual graphics and audio

- 

Affordance Framework

William W. Gaver. 1991. Technology affordances. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '91), Scott P. Robertson, Gary M. Olson, and Judith S. Olson (Eds.). ACM, New York, NY, USA, 79-84.

Affordance Led Design   Make affordances perceivable

  Provide visual, haptic, tactile, auditory cues

  Affordance Led Usability   Give feedback   Provide constraints   Use natural mapping   Use good cognitive model

Example: AR Chemistry

  Tangible AR chemistry education (Fjeld) Fjeld, M., Juchli, P., and Voegtli, B. M. 2003. Chemistry education: A tangible interaction

approach. Proceedings of INTERACT 2003, September 1st -5th 2003, Zurich, Switzerland.

Input Devices

  Form informs function and use

Picking up an Atom

AR Interaction Metaphors

  Interface Components  Physical components  Display elements

- Visual/audio   Interaction metaphors

Physical Elements

Display Elements Interaction

Metaphor Input Output

AR Design Elements

Interface Design Path

1/ Prototype Demonstration

2/ Adoption of Interaction Techniques from other interface metaphors

3/ Development of new interface metaphors appropriate to the medium

4/ Development of formal theoretical models for predicting and modeling user actions

Desktop WIMP

Virtual Reality

Augmented Reality

Interface metaphors   Designed to be similar to a physical entity but also has own

properties   e.g. desktop metaphor, search engine

  Exploit user’s familiar knowledge, helping them to understand ‘the unfamiliar’

  Conjures up the essence of the unfamiliar activity, enabling users to leverage of this to understand more aspects of the unfamiliar functionality

  People find it easier to learn and talk about what they are doing at the computer interface in terms familiar to them

Example: The spreadsheet   Analogous to ledger

sheet   Interactive and

computational   Easy to understand   Greatly extending

what accountants and others could do

www.bricklin.com/history/refcards.htm

Why was it so good?   It was simple, clear, and obvious to the users how to

use the application and what it could do   “it is just a tool to allow others to work out their

ideas and reduce the tedium of repeating the same calculations.”

  capitalized on user’s familiarity with ledger sheets   Got the computer to perform a range of different

calculations in response to user input

The Star interface

Benefits of interface metaphors   Makes learning new systems easier   Helps users understand the underlying

conceptual model   Can be innovative and enable the realm of

computers and their applications to be made more accessible to a greater diversity of users

Problems with interface metaphors (Nielson, 1990)

  Break conventional and cultural rules   e.g., recycle bin placed on desktop

  Can constrain designers in the way they conceptualize a problem   Conflict with design principles   Forces users to only understand the system in terms of the

metaphor   Designers can inadvertently use bad existing designs and transfer

the bad parts over   Limits designers’ imagination with new conceptual models

Microsoft Bob

  PSDoom – killing processes

Back to the Real World

  AR overcomes limitation of TUIs   enhance display possibilities  merge task/display space   provide public and private views

  TUI + AR = Tangible AR   Apply TUI methods to AR interface design

Tangible AR Design Principles   Tangible AR Interfaces use TUI principles

  Physical controllers for moving virtual content   Support for spatial 3D interaction techniques   Time and space multiplexed interaction   Support for multi-handed interaction  Match object affordances to task requirements   Support parallel activity with multiple objects   Allow collaboration between multiple users