the challenges and potential of end-user gesture customization

The Challenges and Potential of End-User Gesture Customization

Uran Oh1 and Leah Findlater21 Department of Computer Science2 College of Information StudiesUniversity of Maryland, College Park

�

[email protected] | [email protected]

http://www.cs.umd.edu/hcil/

Touchscreen gestures are widely used…Who designs these gestures?Design experts.

Apple’s touchpad gestures

1) Tools for supporting designers (developers)to create gestures with ease

Previous Research:

A figure from Gesture Coder

MAGIC:[Ashbrook et al. 2010]

Proton++:[Kin et al. 2012]

Gesture Coder:[Lü et al. 2012]

2) Methods for creating a gesture set that are intuitive and guessable by a wide range of users

[Wobbrock et al. 2009], [Kray et al. 2010], [Ruiz et al. 2011]

Previous Research:

A figure from [Wobbrock et al. 2009]

(2) Methods for creating a gesture set that are intuitive and guessable by a wide range of users


Previous research:


Our focus: Supporting end-users

Personal gestures for a single user

(2) Methods for creating a gesture set that are intuitive and guessable by a wide range of users


Previous research:


Our focus: Supporting end-users

Personal gestures for a single userWhy?

MemorabilityEfficiency

Accessibility

Potential Advantages of Self-defined Gestures…

[Nacenta et al. 2013]

Memorability

�

Self-defined gestures improve memorability over predefined gestures

[Ouyang et al. 2012]

Efficiency

�

Gestural shortcuts can be used as an efficient mean of accessing information

Accessibility

[Anthony et al. 2013]

�

Customized gestures may improve accessibilityfor people with physical disabilities

Our Goal: To investigate the feasibility of end-user gesture creation

�

Our Goal: To investigate the feasibility of end-user gesture creation

�

How do typical users create gestures?

What are the challenges therein?

How can we support the process?

v

Task 1 Task 2 Task 3Task 2

Open-EndedGesture Creation

Action-SpecificGesture Creation

Saliency ofGesture Features

Study With Three Tasks

Controlled lab study - 20 participants (age from 20 to 35,

M=29.3)- Prior experience with touchscreen devices- Single one-hour session with 3 tasks- Think-aloud protocol

Study Method

�

Apparatus- Samsung Galaxy Tab

2 (10.1’’ running Android 4.0.4)

v

Task 1

Task 2 Task 3Task 2




Q. Are users able to create new gestures easily?If not, what are the barriers?

�

Task 1: Open-ended Gesture Creation

“Create as many gestures as possible”

�

“Create as many gestures as possible”Task 1: Open-ended Gesture Creation

• For any purpose• Any number of strokes, fingers,

hands • As long as they are:

easy to draw, easy to remember, distinguishable

�


“Create as many gestures as possible”

12.2 gestures created on average (SD = 8.1, range of 5 to 36)

Gestures Created

p3

Total number of gestures and the number of arbitrary gestures are correlated

(Pearson’s r=.47, p=.037)


p5

12.2 gestures created on average (SD = 8.1, range of 5 to 36)

Gestures Created

p3 p5

Total number of gestures and the number of arbitrary gestures are correlated

(Pearson’s r=.47, p=.037)


p5

Tendency to focus on the familiar“I just thought of gestures my tablet PC had.” (P1)“These gestures are all I use, I cannot be more creative” (P8)

Difficulties Creating Gestures

�

Opaque nature of gesture recognizer“Can I use all fingers?” (P2)


v

Task 1

Task 2 Task 3Task 2




A. Users felt difficulty in creating new gesturesBetter understanding of recognizer is needed

23

Task 1

Task 2

Task 3

Task 2




Q. What is a “good gesture” to end-users?How is it different from recognizer’s perspective?

Task 2: Action-Specific Gesture Creation

Brainstorm gesturesper action

12 Specific ActionsZoom-inZoom-outRotateCopyCutPasteSelect-singleSelect-multiplePreviousNextCall-MomLaunch a web-browser

12 Specific ActionsZoom-inZoom-outRotateCopyCutPasteSelect-singleSelect-multiplePreviousNextCall-MomLaunch a web-browser




Compose custom set of gestures, one per

action





action

Create training examples

(4 per selected gesture)




action



Rate satisfaction with the custom gesture

set



action



Rate satisfaction with the custom gesture

set

Test recognition accuracy with $N

recognizer

Initial example

Training examples


Generally Preferred

Accurate

Familiar

Simple/Easy

Intuitive/Natural/Obvious

0 5 10 15 20 25 30

11.34

12.18

15.55

22.69

27.73

Percentage of Gestures (%)

Reasons for selecting a gesture for custom set

Others reasons: Generally preferred, fast, consistent, easy to remember, etc.


Need for improvement

Participants gave up the opportunity to edit their gesture set to make improvements


Only two participants were fully satisfied

( M=5.3, SD = 1.1 where 1=negative, 7=positive)

Inability to improve gesture sets

Low Recognition Potential of the Custom Sets$N recognizer (default setting) with 5-fold cross validation

1 2 3 40.7

0.8

0.9

Number of Training Examples

Reco

gniti

on A

c-cu

racy

�

76–88% accuracy depending on amount of training


35

Task 2

Task 3

Task 2



Customized set can be improved for both user’s and recognizer’s perspectiveA.

Task 1


36

Task 2

Task 3

Task 2



What features do users rely on to distinguish between gestures?Q.

Task 1


Gesture Features Judged

Orientation

Very slow Very fast slow fast moderate

Scale

Aspect Ratio

Speed

Task 3: Saliency of Gesture Features

Curviness

Pattern Repetition

6 features from Rubine’s recognizer [Rubine. 1991]

Gesture Features Judged

Orientation

Scale

Aspect Ratio


Curviness

Pattern Repetition

Finger Count

Stroke Count

Stroke Order

3 touchscreen features



Speed

Orientation

Scale

Aspect Ratio


Curviness

Pattern Repetition

Finger Count

Stroke Count

Stroke Order

3 touchscreen features


“Rank the distinguishability of 9 features”


Speed

Objective features are more distinguishableFeatures that can be consistently interpreted/manipulated are considered distinguishable

“Even if the same person is performing the gesture, it might not have the same speed and size” (P7)

More distinctive

Very fast

SpeedScale

Pattern Repetiti

on

Aspect Ratio

Curviness

Orientatio

n

Stroke Order

Stroke co

unt

Finger count


43

Task 2

Task 3

Task 2



A.

Task 1


Number of fingers/strokes, stroke order aredistinguishable than speed or size

SummaryCreating new gestures is hard for end-users• Tendency to focus on the familiar• Opaque nature of gesture recognizer

�

Objective features are more distinguishable• Finger/stroke count, stroke order are

more distinguishable than speed and scale

Quality of gesture sets can be improved• Users are not fully satisfied with their

gesture sets• Low recognition potential

Memorability

Efficiency

Accessibility�

Potential Benefits of Allowing End-User Customization

Take-away Message

Systematic Support is Needed for End-User

Customization

Future WorkMixed-initiative support for customization

Feedback

EditsTrain

System Gesture set User

47

The Challenges and Potential of End-User Gesture Customization

Uran Oh1 and Leah Findlater21 Department of Computer Science2 College of Information StudiesUniversity of Maryland, College Park

�

[email protected] | [email protected]

Thank you for listening

Questions?

http://www.cs.umd.edu/hcil/

the challenges and potential of end-user gesture customization

Documents