Introduction to Wearables

Mark Billinghurst HIT Lab NZ

December 2014

Room Desk Lap Hand Head

Wearable Computing ▪  Computer on the body that is: ▪  Always on ▪  Always accessible ▪  Always connected

▪  Other attributes ▪  Augmenting user actions ▪  Aware of user and surroundings

The Ideal Wearable ▪  Persists and Provides Constant Access: Designed

for everyday and continuous user over a lifetime. ▪  Senses and Models Context: Observes and

models the users and environment. ▪  Augments and Mediates: Information support for

the user in both the physical and virtual realities. ▪  Interacts Seamlessly: Adapts its input and output

modalities to those most appropriate at the time.

Starner, T. E. (1999). Wearable computing and contextual awareness (Doctoral dissertation, Massachusetts Institute of Technology).

Wearable Attributes

▪  fafds

History

History of Wearables ▪  1960-90: Early Exploration ▪  Custom build devices

▪  1990 - 2000: Academic, Military Research ▪  MIT, CMU, Georgia Tech, EPFL, etc ▪  1997: ISWC conference starts

▪  1995 – 2005+: First Commercial Uses ▪  Niche industry applications, Military

▪  2010 - : Second Wave of Wearables ▪  Consumer applications, Head Worn

The Gamblers

▪  Timing device for roulette prediction ▪  Card counting hardware (toe input)

Ed Thorp (1961)

Thorp, E. O. (1998, October). The invention of the first wearable computer. In Wearable Computers, 1998. Second International Symposium on (pp. 4-8). IEEE.

Belt computer

Shoe Input

Glasses Display

Keith Taft (1972)

The Hackers (1980’s - )

!  MIT Media Lab – Wearable Computing (1993)

http://www.media.mit.edu/wearables/

Enabling Technologies (1989+) ▪  Private Eye Display (Reflection Tech.) ▪  720 x 280 dipslay ▪  Red LED ▪  Vibrating mirror

▪  Twiddler (Handykey) ▪  Chording keypad ▪  Mouse emulation

MIT Tin Lizzy (1993) ▪  General Purpose Wearable ▪  Doug Platt, Thad Starner ▪  150 MHz Pentium CPU ▪  32-64 Mb RAM, 6 Gb HDD ▪  VGA display ▪  Cellular modem

http://www.media.mit.edu/wearables/lizzy/lizzy/index.html

Prototype Applications ▪  Remembrance Agent ▪  Rhodes (97)

▪  Augmented Reality ▪  Feiner (97), Thomas (98)

▪  Remote Collaboration ▪  Garner (97), Kraut (96)

▪  Maintenance/Factory ▪  Caudell (92), Thompson (97)

Mobile AR: Touring Machine (1997)

▪  University of Columbia ▪  Feiner, MacIntyre, Höllerer, Webster

▪  Combines ▪  See through head mounted display ▪  GPS tracking ▪  Orientation sensor ▪  Backpack PC (custom) ▪  Tablet input

Feiner, S., MacIntyre, B., Höllerer, T., & Webster, A. (1997). A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment. Personal Technologies, 1(4), 208-217.

MARS View ▪  Virtual tags overlaid on the real world ▪  “Information in place”

PCI 3D Graphics Board

Hard Drive

Serial Ports

CPU

PC104 Sound Card

PC104 PCMCIA

GPS Antenna

RTK correction Antenna

HMD Controller

Tracker Controller

DC to DC Converter

Battery

Wearable Computer

GPS RTK correction

Radio

Example self-built working solution with PCI-based 3D graphics

Columbia Touring Machine

Mobile AR - Hardware

Early Commercial Systems ▪  Xybernaut (1996 - 2007) ▪  Belt worn, HMD, 200 MHz

▪  ViA (1996 – 2001) ▪  Belt worn, Audio Interface ▪  700 MHz Crusoe

▪  Symbol (1998 – 2006) ▪  Wrist worn computer ▪  Finger scanner

Symbol WWC 1000 (1998 - )

!  Wrist worn wearable + finger barcode scanner !  $3500 USD, current price $1000 ! Over 30K sold in first 2 years, still selling (>100k units?) !  First widely deployed wearable computer

Reasons For Success !  Well defined large market niche

!  Stock pickers with holster scanners

!  Significant usability/ergonomics effort ! Over 40,000 hours user testing

!  Provided significant performance improvement ! Met user needs, solved existing problems

!  Addressed social factors !  Company with substantial R+D resources

Stein, R., Ferrero, S., Hetfield, M., Quinn, A., & Krichever, M. (1998, October). Development of a commercially successful wearable data collection system. In Wearable Computers, 1998. Digest of Papers. Second International Symposium on (pp. 18-24). IEEE.

Second Gen. Commercial Systems !  Recon (2010 - )

! Head worn displays for sports !  Ski goggle display !  Investment from Intel (2013)

!  Google (2011 - ) ! Google Glass ! Consumer focus

Recon MOD/MOD live (2010 - ) !  http://www.reconinstruments.com !  Ski goggle wearable

! Niche market focus !  Show speed, maps, phone calls etc !  $199/$299 accessory ! GPS location, orientation, sensor output !  Android platform/SDK

!  Sales: > 50K units sold

Use Case !  While skiing

show: ! maps, !  speed, !  altitude !  phone calls !  text messages

Google Glass (2011 - )

Other HWCs ▪  Vuzix M-100 ▪  $999, professional

▪  Recon Jet ▪  $600, more sensors, sports

▪ Opinvent ▪  500 Euro, multi-view mode

▪ Motorola Golden-i ▪  Rugged, remote assistance

dsfh

Summary !  First wearable companies

!  Targeting niche markets !  Expensive/poorly designed solutions ! Mostly low sales (< 10,000)

!  Current generation !  First general consumer wearable (Glass, others) !  Bigger niche markets (skiing, sports) – > 50K+ sales !  Lower costs/better design

Market Analysis

Gartner - Hype Cycle

!  5-10 years to widespread commercial acceptance

5 – 10 years to widespread commercialization

Projected Market - 1

Total Number Wearable Devices

Wide Difference in Market Size

Key Companies

Summary !  Wearables reach mass market over 5-10 years !  Analysts disagree on projected sales

! Maybe 20 million devices shipped/year by 2018 ! Up to $11 Billion market size by 2018

!  Wearable computers small cf to other technology !  Smart glass around 10% of all wearable sales !  All wearable devices < 10% of smart phone sales

Experience Design

“The product is no longer the basis of value. The experience is.”

Venkat Ramaswamy The Future of Competition.

Using the N-gage

SideTalking

!  www.sidetalkin.com

Experience Design is All About You

!  Users should be involved throughout the Design Process

!  Consider all the needs of the user

Glass User Experience !  Truly Wearable Computing

!  Less than 46 ounces

!  Hands-free Information Access !  Voice interaction, Ego-vision camera

!  Intuitive User Interface !  Touch, Gesture, Speech, Head Motion

!  Access to all Google Services !  Map, Search, Location, Messaging, Email, etc

Living Heads Up vs. Heads Down

Lost in Screen Space

PHYSICAL

Physical Elements !  Interface design !  Input techniques !  Interaction metaphors !  Wearability/Ergonomics

Design For Device

!  Simple, relevant information !  Complement existing devices

Typical Usage Times

Consider Your User !  Wearable User

!  Probably Mobile ! One/no hand interaction !  Short application use ! Need to be able to multitask ! Use in outdoor or indoor environment ! Want to enhance interaction with real world

It's  like  a  rear  view  mirror    

Don't  overload  the  user.  S2ck  to  the  absolutely  essen2al,  avoid  long  

interac2ons.  Be  explicit.      

Micro  Interac3ons  

The  posi2on  of  the  display  and  limited  input  ability  makes  longer  interac2ons  less  comfortable.  

 Using  it  shouldn’t  take  longer  than  taking  out  your  phone.  

Micro-Interactions

▪  On mobiles people split attention between display and real world

Time Looking at Screen

Oulasvirta, A. (2005). The fragmentation of attention in mobile interaction, and what to do with it. interactions, 12(6), 16-18.

Design for MicroInteractions ▪  Design interaction less than a few seconds

!  Tiny bursts of interaction !  One task per interaction !  One input per interaction

▪  Benefits !  Use limited input !  Minimize interruptions !  Reduce attention fragmentation

Do one thing at a time

Make it Glanceable

!  Seek to rigorously reduce information density. !  Design for recognition, not reading.

Bad Good

Reduce the number of info chunks

Reducing the total # of information chunks will increase the glanceability of your design.

1

2

3

1

2

3

4

5 (6)

Single Interactions Faster than 4 s

1

2

3

1

2

3

4

5 (6)

Eye Movements: ~ 2 secs

Eye Movements: ~ 0.9 sec

Test the glanceability of your design ✓

Don’t Get in the Way

!  Enhance, not replace, real world interaction

Keep it Relevant

!  Information at the right time and place

✓ Test your design indoors + outdoors

Interface Guidelines ▪  Don’t design a mobile app !  Design for emotion ▪  Make it glanceable !  Do one thing at a time !  Reduce number of information chunks ▪  Design for indoor and outdoor use

Physical Input !  Interaction on the go

! Must support mobile input

!  Fatigue !  “Gorrilla” Arm from free-hand input

!  Comfort !  People want to do small gestures by waist

!  Socially Acceptable ! Do I look silly doing this?

Twiddler Input

▪  Chording or multi-tap input ▪  Possible to achieve 40 - 60 wpm after 30+ hours ▪  Chording input about 50% faster than multi-tap ▪  cf 20 wpm on T9, or 60+ wpm for QWERTY

Lyons, K., Starner, T., Plaisted, D., Fusia, J., Lyons, A., Drew, A., & Looney, E. W. (2004, April). Twiddler typing: One-handed chording text entry for mobile phones. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 671-678). ACM.

Virtual Keyboards

▪  In air text input ▪  Virtual QWERTY keyboard up to 20 wpm ▪  Word Gesture up to 28 wpm

▪  Handwriting around 20-30 wpm

A. Markussen, et. al. Vulture: A Mid-Air Word-Gesture Keyboard (CHI 2014)

Unobtrusive Input Devices

▪  GestureWrist ▪  Capacitive sensing ▪  Change signal depending on hand shape

Rekimoto, J. (2001). Gesturewrist and gesturepad: Unobtrusive wearable interaction devices. In Wearable Computers, 2001. Proceedings. Fifth International Symposium on (pp. 21-27). IEEE.

GesturePad

Skinput

Using EMG to detect muscle activity

Tan, D., Morris, D., & Saponas, T. S. (2010). Interfaces on the go. XRDS: Crossroads, The ACM Magazine for Students, 16(4), 30-34.

Types of Head Mounted Displays

Occluded See-thru

Multiplexed

See-through Thin Displays

▪  Waveguide techniques for thin see-through displays ▪  Wider FOV, AR applications, Social acceptability

Opinvent Ora

Lumus

Where to put Wearables?

▪  Places for unobtrusive wearable technology

Gemperle, F., Kasabach, C., Stivoric, J., Bauer, M., & Martin, R. (1998, October). Design for wearability. In Wearable Computers, 1998. Digest of Papers. Second International Symposium on (pp. 116-122). IEEE.

Where to Place Trackpad?

▪  User study 25 people different postures ▪  Front of thigh most preferred, torso/upper arm worst

Thomas, Bruce, et al. "Determination of placement of a body-attached mouse as a pointing input device for wearable computers.” ISWC 1999. IEEE Computer Society, 1999.

Where do users want Wearables?

29% on clothing 28% on wrist 12% on Glasses

SOCIAL/CULTURAL

TAT Augmented ID

How is the User Perceived?

The Glasshole Effect

Social Interaction

!  For mobiles, location and audience have a significant impact on the type of gestures people will be willing to perform.

Professional vs. Everyday Use

!  Everyday acceptance ! Only 12% US adults said they would use AR glasses !  20% handheld AR users experienced social issues !  40% neutral/uncomfortable with public gestures

!  Professional Use !  85% Doctors said they would use AR glass in job !  < 25% neutral/uncomfortable with work gestures

COGNITIVE

Cognitive Load and Interfaces

A Typical Work Day

Which is the most cognitively demanding?

Cognitive Interference ▪  Structural interference ▪  Two or more tasks compete for limited

resources of a peripheral system -  eg two cognitive processes needing vision

▪  Capacity interference ▪  Total available central processing overwhelmed

by multiple concurrent tasks -  eg trying to add and count at same time

Resource Competition Framework

▪  Mobility tasks compete for cognitive resources with other tasks ▪  the most important given higher priority

▪  RCF is a method for analyzing this, based on: ▪  task analysis ▪  modelling cognitive resources ▪  a resource approach to attention

Oulasvirta, A., Tamminen, S., Roto, V., & Kuorelahti, J. (2005, April). Interaction in 4-second bursts: the fragmented nature of attentional resources in mobile HCI. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 919-928). ACM.

Cognitive Resources & Limitations

Application of RCF

Busy street > Escalator > Café > Laboratory. But if you made Wayfinding, Path Planning, Estimating

Time to Target, Collision Avoidance easier?

Handling Interruptions

▪  Gradually increase engagement ▪  Reduce attention load

Receiving SMS on Glass

“Bing”

Tap Swipe

Glass

Show Message Start Reply User

Look Up

Say Reply

Nomadic Radio (2000)

▪  Spatial audio wearable interface Sawhney, N., & Schmandt, C. (2000). Nomadic radio: speech and audio interaction for contextual messaging in nomadic environments. ACM transactions on Computer-Human interaction (TOCHI), 7(3), 353-383.

Spatial Audio Metaphor

Messages arranged depending on time of day

Notification Interruptions

▪  Dynamic scaling of incoming message based on interruptibility of the user ▪  Busy = silence ▪  Availble = preview

Use Cases

Ideal Applications !  Use cases that require:

! Hands-free interaction ! Mobile information access ! Constant access to information !  Supporting activity in real world !  Low likelihood of social issues !  Enhanced view of reality

Wearable Use Cases

Smart Watch Consumer Insights/Primary Uses

Personal assistance Health & Fitness Personal safety

Communication Smart home access Near Field Communication

(NFC)

Social  ac3on    First-­‐person  journalist  Tim  Pool    broadcasts  an  in2mate  view    of  Istanbul  protests.                'I  want  to  show  you  what  it's  like  to  be  there  as  best  I  can,  even  if  that  ends  with  me  running  full-­‐speed  into  a  cafe  and  rubbing  lemons  all  over  my  face  a<er  being  tear-­‐gassed'        

CityViewAR

!  Using AR to visualize Christchurch city buildings !  3D models of buildings, 2D images, text, panoramas !  AR View, Map view, List view !  Available on Android market

CityViewAR on Glass

!  AR overlay of virtual buildings in Christchurch

Financial Services Applications !  Which Financial Services task could benefit from

using wearables? !  always on, connected, accessible

!  Match task requirements to wearable attributes ! Hands-busy, constant information access, user

monitoring, mobile, etc

!  Key question: can this task be done on a mobile phone instead of wearable?

Financial Applications !  Possible types of Financial Services experiences

! Customer Service ! Consumer Banking !  Finance Trading !  Remote Collaboration

CUSTOMER SERVICE

Customer Service !  Wearables provide

! Constant access to information ! Mobile access – move agent from behind counter !  Improved face to face customer experience

-  Customer face recognition -  Eye to eye contact

Example: Virgin Atlantic

!  Virgin Atlantic trialing Glass for customer check in !  Features

!  Agent greets customer curb-side, start check-in process !  Provide customer details, personalized service !  Document verification – camera scanning of boarding pass

!  Advantages !  Focus attention on customer ! Moves agent to customer

-  Earlier engagement

!  Reduces technology barrier between agent and customer

-  Hide behind computer/desk

!  Provides personalized service -  Name, flight details, weather, diet,

translation services, etc

“The trial was a huge success with positive feedback from both our staff and customers on the usage of wearable technology” !  Key findings

!  Only Google Glass permitted the agent to maintain eye contact showing they were engaged and interesting in helping.

!  Some passengers were taken aback initially by Glass wearing concierges, but, on the whole, passengers responded well.

!  Technical challenges to overcome -  Short battery life, camera resolution, wifi issues

CONSUMER BANKING

Consumer Banking !  Use Glass to provide mobile finance services

!  Bank locator !  In-store payments ! Currency conversion !  Receipt capture (using camera) !  Account balance checking !  Etc

!  Some banks porting their mobile apps to Glass

Example: Westpac !  Ported mobile finance app Cash Tank to Glass !  Features

! Check balance !  Transfer funds !  Find nearest ATM/bank branch ! Get alerts and notifications

!  Trialed in NZ and Australia

Demo Video

!  https://www.youtube.com/watch?v=6qX25cE7Bww

Other Examples !  CaixaBank !  Well Fargo !  Discover !  Saudi Investment Bank ATM finder !  FISERV Mobile payments on Glass !  TMG See2Pay small-dollar payments on Glass

!  http://blog.dwolla.com/dwolla-now-available-on-google-glass/

See2pay Demo

!  https://www.youtube.com/watch?v=D-3UKp0a4nU

CiaxaBank Glass Application

Branch finder (AR), currency converter, contact details

FINANCE TRADING

Finance Trading !  Currently most finance trading is confined to

fixed desks/location !  Wearables could allow managers to maintain

awareness while away from trading floor !  Provides constant access to markets !  Alerts where ever they are ! Constant monitoring of individual funds !  Etc.

Deskbound to Mobile

COLLABORATION

Google Glass

▪  Wearable computer – Ego-Vision view ▪  camera + processing + display + connectivity

Example – Vipaar Telemedicine

!  Vipaar + UAB - http://www.vipaar.com !  Endoscopic view streamed remotely !  Remote expert adds hands – viewed in Glass

VIPAAR Demo

https://www.youtube.com/watch?v=aTOoBwfqBe0

GoToAssist !  http://www.gotoassist.com/ !  Remote collaboration platform

!  Screen sharing !  File transfer ! Chat, Annotation

!  SeeIt platform supports Glass

SeeIt Glass Demo

!  http://www.citrix.com/tv/#videos/10065

Research Problems

Challenges for the Future (2001) ▪  Privacy ▪  Power use ▪  Networking ▪  Collaboration ▪  Heat dissipation ▪  Interface design ▪  Intellectual tools ▪  Augmented Reality systems

Starner, T. (2001). The challenges of wearable computing: Part 1. IEEE Micro,21(4), 44-52. Starner, T. (2001). The challenges of wearable computing: Part 2. IEEE Micro,21(4), 54-67.

Gesture Interaction With Glass ▪  3 Gear Systems ▪  Hand tracking

▪  Hand data sent to glass ▪  Wifi networking ▪  Hand joint position ▪  AR application rendering ▪  Vuforia tracking

Performance

▪  Full 3d hand model input ▪  10 - 15 fps tracking, 1 cm fingertip resolution

● Meta Gesture Interaction

▪ Depth sensor + Stereo see-through ▪ https://www.spaceglasses.com/

Current Collaboration

▪  First person remote conferencing/hangouts ▪  Limitations

-  Single POV, no spatial cues, no annotations, etc

Sharing Space: Social Panoramas

▪  Capture and share social spaces in real time ▪  Enable remote people to feel like they’re with you

Context Sensing ▪  Using context to manage information ▪  progressive information display as user shows

interest

▪  Context from ▪  Speech ▪  Gaze ▪  Real world

▪  Wearable AR Display

Ajanki, A., Billinghurst, M., Gamper, H., Järvenpää, T., Kandemir, M., Kaski, S., ... & Tossavainen, T. (2011). An augmented reality interface to contextual information. Virtual reality, 15(2-3), 161-173.

Gaze Interaction

AR View

More Information Over Time

Conclusions

Conclusions !  Wearable computing represents a fourth

generation of computing devices !  Google Glass is the first consumer wearable

!  Lightweight, usable, etc

!  A range of wearables will appear in 2014 !  Ecosystem of devices

!  Significant research opportunities exist ! User interaction, displays, social impact

More Information •  Mark Billinghurst

– mark.billinghurst@hitlabnz.org

– @marknb00

•  Website – www.hitlabnz.org