rapid prototyping for augmented reality

RAPID PROTOTYPING FOR AUGMENTED REALITY

Mark Billinghurst [email protected]

April 5th 2016

How do you Design and Test this?

Or this..

Google Glass

View Through Glass

Google Glass Prototyping

http://ed.ted.com/lessons/rapid-prototyping-google-glass-tom-chi

Rapid Prototyping Glass • Q: How long would it take you to make the first working version of the Glass headset display?

• A: One day..

Interacting with Glass

Prototyping Interaction • Q: How long would it take you to prototype a Minority Report interaction technique?

• A: 45 minutes..

Tom Chi’s Prototyping Rules 1.  Find the quickest path to experience 2.  Doing is the best kind of thinking 3.  Use materials that move at the speed of

thought to maximize your rate of learning

The Road to Glass

Early prototyping

DESIGN PROCESSES

Interaction Design Process

Design Thinking - Three Phase Model

http://dschool.stanford.edu/redesigningtheater/the-design-thinking-process/

Elaboration and Reduction

• Elaborate - generate solutions. These are the opportunities • Reduce - decide on the ones worth pursuing • Repeat - elaborate and reduce again on those solutions

Source: Laseau,P. (1980) Graphic Thinking for Architects & Designers. John Wiley and Sons

Elaboration vs. Reduction

• Elaboration: Sketching/Storytelling • Creation, exploration • Quick, conceptual

• Reduction: Prototyping/Testing • Functional •  Idea presentation • Decision making

RAPID PROTOTYPING

How can we quickly prototype Augmented Reality experiences

with little or no coding?

Why Prototype?

▪  Quick visual design ▪  Capture key interactions ▪  Focus on user experience ▪  Communicate design ideas ▪  “Learn by doing/experiencing”

Design/Prototyping Tools

Typical Development Steps

▪  Sketching ▪  Storyboards ▪  UI Mockups ▪  Interaction Flows ▪  Video Prototypes ▪  Interactive Prototypes ▪  Final Native Application

Increased Fidelity & Interactivity

Sketching Sketching is not about drawing It is about design. Sketching is a tool to help you:

•  express •  develop, and •  communicate design ideas

Sketching is part of a process:

•  idea generation, •  design elaboration •  design choices, •  engineering

Figure 41: A Sketch of a Dialogue with a Sketch

The “conversation” between the sketch (right bubble) and the mind (left

bubble). A sketch is created from current knowledge (top arrow). Reading,

or interpreting the resulting representation (bottom arrow), creates new

knowledge. The creation results from what Goldschmidt calls “seeing

that” reasoning, and the extraction of new knowledge results from what

she calls “seeing as.”

sketchrepresentation

mind(new) knowledge

Create(seeing that)

Read(seeing as)

“Sketching is about the activity not the result” – Bill Buxton

25

AR Hockey Concept

https://stereoscopica.wordpress.com/2013/05/29/augmented-reality-hockey-concept/

Sketched Interfaces

▪  Sketch + Powerpoint/Photoshop/Illustrator

Sketched Concepts

http://www.objecttheory.com/

Example Storyboard

Example Storyboard p.2

Persona + Storyboard

Paper Prototyping (Low Fidelity)

Quick and simple means of sketching interfaces Use office materials Easier to criticize, quick to change Creative process (develop in team) Can also use for usability test (focus on interaction flow) Used a lot to test out concepts before real design begins.

Paper Prototyping

The Basic Materials

• Post-its • 5x8 in. index cards • Scissors, X-acto knives • Overhead transparencies • Large, heavy, white paper (11 x 17) • Tape, stick glue, correction tape • Pens & markers (many colors & sizes)

Physical Prototype

• Early ideation • Think through ideas • Force you to visualize how things come together • Communicate ideas to others to inspire new designs • Active brainstorming

Why is Sketching/Paper Prototyping Useful?

Paper Prototypes Pros and Cons • Good

• Quick and dirty • No software needed • Very inclusive

• Bad • Not quick enough •  Too dirty (lack context and scale) • Hard to share among team • Still needs a separate documentation step

From Sketches to Working Systems Early design

Late design

Brainstorm different ideas and representations

Choose a representation Rough out interface style

Multitude of sketches

Sketch variations and details

Sketch or low fidelity prototypes Task centered walkthrough and redesign

Fine tune interface, screen design Heuristic evaluation and redesign

Usability testing and redesign

Low to medium fidelity prototypes

Limited field testing

Alpha/Beta tests

High fidelity prototypes

Working systems

From Sketches to Prototypes •  Sketches: early ideation stages of design •  Prototypes: capturing /detailing the actual design

Prototyping

• Create physical form of ideas • Allow people to experience and interact with them

• Why Prototyping? • Empathy gaining- deepen understanding of design space • Exploration – build to think • Testing – test solutions with end users •  Inspiration – help others catch your vision

Sketch vs. Prototype Sketch PrototypeInvite A)endSuggest DescribeExplore RefineQues;on AnswerPropose TestProvoke ResolveTenta;ve,noncommi)al SpecificDepic;on

The primary differences are in the intent

Sketching to Prototype

Wireframe

• It’s about • Functional specs • Navigational systems • Functionality and layout • Notes about the intended functionality • How interface elements work together • Lack of typographic style, colour or graphics

• Leaving room for the design to be created

Wireframes

Mockup

•  It’s about •  Look and feel •  Build on the wireframe with

graphics and polish • May adjust layout slightly but stays

within the general guide of the wireframe

FunMe Mobile AR Wireframe/Mockup

http://www.yunnuocheng.com/funme/

GlassSim – http://glasssim.com/

▪  Simulate the view through Google Glass ▪  Multiple card templates

GlassSim Card Builder ▪  Use HTML for card details ▪  Multiple templates ▪  Change background ▪  Own image ▪  Camera view

GlassSim Samples

Application Storyboard

▪  http://dsky9.com/glassfaq/google-glass-storyboard-template-download/

Application Flow

Acting/Role Playing

Wireframe/Mockup Pros and Cons ▪  Positives ▪  Good for documenting screens ▪  Can show application flow

▪  Negatives ▪  No interactivity/transitions ▪  Can’t be used for testing ▪  Can’t deploy on device ▪  Can be time consuming to create

Transitions

▪ Series of still photos in a movie format. ▪ Demonstrates the experience of the product ▪ Discover where concept needs fleshing out. ▪ Communicate experience and interface ▪ You can use whatever tools, from Flash to iMovie.

Video Sketching

UI Concept Movies

Video Sketch Pros and Cons • Good

• Excellent communication tool • Shows interactions • High fidelity concept

• Bad • No interaction • Difficult to use for user testing • May not be able to implement vision •  Too good to be true

EXAMPLE: SKETCH TO VIDEO

AR Hockey

• Concept – Air Hockey in the real world •  Mix materiality and real surfaces with digital forms •  React to real body movement and player interaction


AR Hockey Concept


Sketched Projected Content

Game Setup

Paper/Experience Prototypes

• Having people act out the game

Physical Mockup

Concept Video

https://vimeo.com/79285725

WORKING PROTOTYPES

Interactive Sketching

• Pop App • Pop - https://popapp.in/ • Combining sketching and interactivity on mobiles • Take pictures of sketches, link pictures together

Using Pop

Interactive Wireframing ▪  Developing interactive interfaces/wireframes ▪  Transitions, user feedback, interface design

▪  Web based tools ▪  UXpin - http://www.uxpin.com/ ▪  proto.io - http://www.proto.io/

▪  Native tools ▪  Justinmind - http://www.justinmind.com/ ▪  Axure - http://www.axure.com/

Proto.io - http://www.proto.io/ ▪  Web based mobile prototyping tool ▪  Features ▪  Prototype for multiple devices ▪  Gesture input, touch events, animations ▪  Share with collaborators ▪  Test on device

Proto.io - Interface

Limitations ▪  No access to sensor data ▪  Camera, orientation sensor

▪  No multimedia playback ▪  Audio, video

▪  Simple transitions ▪  No conditional logic

▪  No networking

Wireframe vs. Prototype vs. Mockup • Wireframe

•  Low fidelity representation of design • What UI elements, where UI are placed

• Mockup • High fidelity static design •  Visual design draft

• Prototype • Medium to high fidelity •  Supports user interaction

Interactive Coding - Processing ▪ Programming tool for Artists/Designers ▪ http://processing.org ▪ Easy to code, Free, Open source, Java based ▪ 2D, 3D, audio/video support

▪ Processing For Android ▪ http://wiki.processing.org/w/Android ▪ Strong Android support, builds .apk file

Processing + Augmented Reality • Support for Augmented Reality

•  Camera input •  GPS/Sensor input •  3D Graphics

• Books •  Prototyping Augmented Reality (Mullins) •  Rapid Android Development (Sauter)

• Software •  Ketai Library - http://ketai.org/

•  Android library support – all sensors on Android devices

http://toxiclibs.org/

Other Tools ▪  Wireframing ▪  pidoco ▪  FluidUI

▪  Rapid Development ▪  Phone Gap ▪  AppMachine

▪  Interactive ▪  App Inventor ▪  WearScript

EXAMPLE: CONCEPT TO DEMO

NASA Hololens Concept Demo

• Vision: Work on Mars from your office • Story and sketches based on vision • Led to working Demo

Chesley Bonestell (1940s)

Hololens Story

HoloLens Concept Sketches

Final NASA HoloLens OnSight Demo

https://www.youtube.com/watch?v=o-GP3Kx6-CE

HARDWARE PROTOTYPING

Build Your Own Wearable

▪  MyVu display + phone + sensors

Beady-i

▪  http://www.instructables.com/id/DIY-Google-Glasses-AKA-the-Beady-i/

Physical Input Devices ▪  Can we develop unobtrusive input

devices ? ▪  Reduce need for speech, touch pad input ▪  Socially more acceptable

▪  Examples ▪  Ring, ▪  pendant, ▪  bracelet, ▪  gloves, etc

Prototyping Platform

Arduino Kit Bluetooth Shield Google Glass

Example: Glove Input

▪ Buttons on fingertips ▪ Map touches to commands

Example: Ring Input

▪ Touch strip, button, accelerometer ▪ Tap, swipe, flick actions

How it works

Bracelet

Armband

Gloves

1,2,3,4

Values/output

CASE STUDY

ITERATIVE DESIGN PROCESS

Design(redesign)

PrototypeTestDesignwithUsers(Note

problems)Evaluate(FixIssues)

MOBILE AUGMENTED REALITY FOR SPATIAL NAVIGATION

Sharon Brosnan Bachelor of Science in Digital Media Design

http://www.sharonbrosnan.com/portfolio/mobile-augmented-reality.php

BUNRATTY FOLK PARK

• Irish visitor attraction run by Shannon Heritage • 19th century life is recreated • Buildings from the mid-west have been relocated to the 26-land surrounding Bunratty Castle

• 30 buildings are set in a rural or village setting there.

AUGMENTED REALITY

In Bunratty Folk Park: •  Allows the visitor to point a camera at an exhibit, the

device recognises its by it’s location and layers digital information on to the display

•  3-dimensional virtual objects can be positioned with real ones on display

•  Leads to dynamic combination of a live camera view and information

NAVIGATIONAL AID

Smartphone Platform Most people carry mobile phones Ideal Augmented Reality Technology •  Global tracking tools •  Wireless communication capabilities •  Location based computing •  Large display for interaction

DESIGNING FOR THE VISITOR

HUMAN CENTRED DESIGN

Goal of the Navigational Aid •  Easy to use, clear and understandable •  Useful to visitors •  Creating interaction between the visitor and the aid

through the user interface

Engage the visitor •  It is necessary to understand the visitor to Bunratty •  Identify visitor motives and goals while going through

the Folk Park.


Understanding the technology and Related Works ! Literature ! Similar Technologies ! Electronic Tours in Museum Settings ! Interactions design in Outdoor Museums

Understanding the User over time ! Observations !  Interviews


Findings • Most visitors do not use the map • Most visitors have mobile phones • Visitors want more information • View the Folk Park at their own pace •  Information should be straight to the point • Large social interaction within groups

FUNCTIONALITY • View Options

• Camera View • Map View • List View

• Sub-Options • Places • Events • Restaurants

! AR Features ! Text Information ! 3D Objects ! 3D Tour Guide ! 3D Placement of Buildings

STORYBOARD


Prototyping and User Testing • Low Fidelity Prototyping

• Sketches • Paper Prototyping • Post-It Prototyping • PowerPoint Prototyping

• High Fidelity Prototyping • Wikitude

INITIAL SKETCHES - Photoshop

Pros: •  Good for idea generation •  Cheap •  Concepts seem feasible Cons: •  Not great feedback gained •  Photoshop not fast enough for making changes

POST IT PROTOTYPING

FirstDraJ

CameraViewwith3D

SecondDraJ ThirdDraJ• Selec;onhighlightedinblue • Homebu)onaddedforeasy

naviga;ontomainmenu

POWERPOINT PROTOTYPING Benefits •  Used for User Testing •  Interactive •  Functionalities work •  Quick •  Easy arrangement of slides

User Testing •  Participants found •  15 minute sessions screen captured •  ‘Talk Allowed’ technique used •  Notes taken •  Post-Interview

WIKITUDE

• Popular AR browser • Mapping • Point of Interest abilities • Multiplatform •  Shows points of interest

• Markers can be selected in and information pop-up appears

FINAL CONCEPT DESIGN

FINAL DESIGN CONCEPT

Key Issues •  Fix issues found in previous sessions

•  Design with guidelines in mind

•  Appealing to the Mental Model •  Icon Design

•  Aesthetic Design •  Colour/Font •  Buttons •  Look

VIDEO PROTOTYPE

" Flexibletoolforcapturingtheuseofaninterface

" Elaboratesimula;onofhowthenaviga;onalaidwillwork

" Doesnotneedtoberealis;cineverydetail

" Givesagoodideaofhowthefinishedsystemwillwork

SUMMARY

Tom Chi’s Prototyping Rules 1.  Find the quickest path to experience 2.  Doing is the best kind of thinking 3.  Use materials that move at the speed of

thought to maximize your rate of learning

Conclusions • Prototyping is one of the most important skills you can have as an ARinterface designer

• There are many tools available: • Sketching, Paper Prototyping, Video, etc

• Always keep the user at the heart of the process • User centered design • User testing/feedback

Consider the Whole User

www.empathiccomputing.org

@marknb00

[email protected]