the challenges and potential of end-user gesture customization

The Challenges and Potential of End-User Gesture Customization

Uran Oh1 and Leah Findlater21 Department of Computer Science2 College of Information StudiesUniversity of Maryland, College Park

�

uranoh@cs.umd.edu | leahkf@umd.edu

Touchscreen gestures are widely used…Who designs these gestures?Design experts.

Apple’s touchpad gestures

1) Tools for supporting designers (developers)to create gestures with ease

Previous Research:

A figure from Gesture Coder

MAGIC:[Ashbrook et al. 2010]

Proton++:[Kin et al. 2012]

Gesture Coder:[Lü et al. 2012]

2) Methods for creating a gesture set that are intuitive and guessable by a wide range of users

[Wobbrock et al. 2009], [Kray et al. 2010], [Ruiz et al. 2011]

Previous Research:

A figure from [Wobbrock et al. 2009]

(2) Methods for creating a gesture set that are intuitive and guessable by a wide range of users

[Wobbrock et al. 2009], [Kray et al. 2010], [Ruiz et al. 2011]

Previous research:

A figure from [Wobbrock et al. 2009]

Our focus: Supporting end-users

Personal gestures for a single user

(2) Methods for creating a gesture set that are intuitive and guessable by a wide range of users

[Wobbrock et al. 2009], [Kray et al. 2010], [Ruiz et al. 2011]

Previous research:

A figure from [Wobbrock et al. 2009]

Our focus: Supporting end-users

Personal gestures for a single userWhy?

MemorabilityEfficiency

Accessibility

Potential Advantages of Self-defined Gestures…

[Nacenta et al. 2013]

Memorability

�

Self-defined gestures improve memorability over predefined gestures

[Ouyang et al. 2012]

Efficiency

�

Gestural shortcuts can be used as an efficient mean of accessing information

Accessibility

[Anthony et al. 2013]

�

Customized gestures may improve accessibilityfor people with physical disabilities

Our Goal: To investigate the feasibility of end-user gesture creation

�

Our Goal: To investigate the feasibility of end-user gesture creation

�

How do typical users create gestures?

What are the challenges therein?

How can we support the process?

v

Task 1 Task 2 Task 3Task 2

Open-EndedGesture Creation

Action-SpecificGesture Creation

Saliency ofGesture Features

Study With Three Tasks

Controlled lab study - 20 participants (age from 20 to 35,

M=29.3)- Prior experience with touchscreen devices- Single one-hour session with 3 tasks- Think-aloud protocol

Study Method

�

Apparatus- Samsung Galaxy Tab

2 (10.1’’ running Android 4.0.4)

v

Task 1

Task 2 Task 3Task 2

Open-EndedGesture Creation

Action-SpecificGesture Creation

Saliency ofGesture Features

Q. Are users able to create new gestures easily?If not, what are the barriers?

�

Task 1: Open-ended Gesture Creation

“Create as many gestures as possible”

�

“Create as many gestures as possible”Task 1: Open-ended Gesture Creation

• For any purpose• Any number of strokes, fingers,

hands • As long as they are:

easy to draw, easy to remember, distinguishable

�

Task 1: Open-ended Gesture Creation

“Create as many gestures as possible”

12.2 gestures created on average (SD = 8.1, range of 5 to 36)

Gestures Created

p3

Total number of gestures and the number of arbitrary gestures are correlated

(Pearson’s r=.47, p=.037)

Task 1: Open-ended Gesture Creation

p5

12.2 gestures created on average (SD = 8.1, range of 5 to 36)

Gestures Created

p3 p5

Total number of gestures and the number of arbitrary gestures are correlated

(Pearson’s r=.47, p=.037)

Task 1: Open-ended Gesture Creation

p5

Tendency to focus on the familiar“I just thought of gestures my tablet PC had.” (P1)“These gestures are all I use, I cannot be more creative” (P8)

Difficulties Creating Gestures

�

Opaque nature of gesture recognizer“Can I use all fingers?” (P2)

Task 1: Open-ended Gesture Creation

v

Task 1

Task 2 Task 3Task 2

Open-EndedGesture Creation

Action-SpecificGesture Creation

Saliency ofGesture Features

A. Users felt difficulty in creating new gesturesBetter understanding of recognizer is needed

23

Task 1

Task 2

Task 3

Task 2

Open-EndedGesture Creation

Action-SpecificGesture Creation

Saliency ofGesture Features

Q. What is a “good gesture” to end-users?How is it different from recognizer’s perspective?

Task 2: Action-Specific Gesture Creation

Brainstorm gesturesper action

12 Specific ActionsZoom-inZoom-outRotateCopyCutPasteSelect-singleSelect-multiplePreviousNextCall-MomLaunch a web-browser

12 Specific ActionsZoom-inZoom-outRotateCopyCutPasteSelect-singleSelect-multiplePreviousNextCall-MomLaunch a web-browser

Task 2: Action-Specific Gesture Creation

Brainstorm gesturesper action

Task 2: Action-Specific Gesture Creation

Compose custom set of gestures, one per

action

Brainstorm gesturesper action

Task 2: Action-Specific Gesture Creation

Compose custom set of gestures, one per

action

Brainstorm gesturesper action

Task 2: Action-Specific Gesture Creation

Compose custom set of gestures, one per

action

Brainstorm gesturesper action

Task 2: Action-Specific Gesture Creation

Brainstorm gesturesper action

Compose custom set of gestures, one per

action

Create training examples

(4 per selected gesture)

Task 2: Action-Specific Gesture Creation

Brainstorm gesturesper action

Compose custom set of gestures, one per

action

Create training examples

(4 per selected gesture)

Rate satisfaction with the custom gesture

set

Brainstorm gesturesper action

Compose custom set of gestures, one per

action

Create training examples

(4 per selected gesture)

Rate satisfaction with the custom gesture

set

Test recognition accuracy with $N

recognizer

Initial example

Training examples

Task 2: Action-Specific Gesture Creation

Generally Preferred

Accurate

Familiar

Simple/Easy

Intuitive/Natural/Obvious

0 5 10 15 20 25 30

11.34

12.18

15.55

22.69

27.73

Percentage of Gestures (%)

Reasons for selecting a gesture for custom set

Others reasons: Generally preferred, fast, consistent, easy to remember, etc.

Task 2: Action-Specific Gesture Creation

Need for improvement

Participants gave up the opportunity to edit their gesture set to make improvements

Task 2: Action-Specific Gesture Creation

Only two participants were fully satisfied

( M=5.3, SD = 1.1 where 1=negative, 7=positive)

Inability to improve gesture sets

Low Recognition Potential of the Custom Sets$N recognizer (default setting) with 5-fold cross validation

1 2 3 40.7

0.8

0.9

Number of Training Examples

Reco

gniti

on A

c-cu

racy

�

76–88% accuracy depending on amount of training

Task 2: Action-Specific Gesture Creation

35

Task 2

Task 3

Task 2

Action-SpecificGesture Creation

Saliency ofGesture Features

Customized set can be improved for both user’s and recognizer’s perspectiveA.

Task 1

Open-EndedGesture Creation

36

Task 2

Task 3

Task 2

Action-SpecificGesture Creation

Saliency ofGesture Features

What features do users rely on to distinguish between gestures?Q.

Task 1

Open-EndedGesture Creation

Gesture Features Judged

Orientation

Very slow Very fast slow fast moderate

Scale

Aspect Ratio

Speed

Task 3: Saliency of Gesture Features

Curviness

Pattern Repetition

6 features from Rubine’s recognizer [Rubine. 1991]

Gesture Features Judged

Orientation

Scale

Aspect Ratio

Task 3: Saliency of Gesture Features

Curviness

Pattern Repetition

Finger Count

Stroke Count

Stroke Order

3 touchscreen features

6 features from Rubine’s recognizer [Rubine. 1991]

Very slow Very fast slow fast moderate

Speed

Orientation

Scale

Aspect Ratio

Task 3: Saliency of Gesture Features

Curviness

Pattern Repetition

Finger Count

Stroke Count

Stroke Order

3 touchscreen features

6 features from Rubine’s recognizer [Rubine. 1991]

“Rank the distinguishability of 9 features”

Very slow Very fast slow fast moderate

Speed

Objective features are more distinguishableFeatures that can be consistently interpreted/manipulated are considered distinguishable

“Even if the same person is performing the gesture, it might not have the same speed and size” (P7)

More distinctive

Very fast

SpeedScale

Pattern Repetiti

on

Aspect Ratio

Curviness

Orientatio

n

Stroke Order

Stroke co

unt

Finger count

Task 3: Saliency of Gesture Features

Objective features are more distinguishableFeatures that can be consistently interpreted/manipulated are considered distinguishable

“Even if the same person is performing the gesture, it might not have the same speed and size” (P7)

More distinctive

Very fast

SpeedScale

Pattern Repetiti

on

Aspect Ratio

Curviness

Orientatio

n

Stroke Order

Stroke co

unt

Finger count

Task 3: Saliency of Gesture Features

Objective features are more distinguishableFeatures that can be consistently interpreted/manipulated are considered distinguishable

“Even if the same person is performing the gesture, it might not have the same speed and size” (P7)

More distinctive

Very fast

SpeedScale

Pattern Repetiti

on

Aspect Ratio

Curviness

Orientatio

n

Stroke Order

Stroke co

unt

Finger count

Task 3: Saliency of Gesture Features

43

Task 2

Task 3

Task 2

Action-SpecificGesture Creation

Saliency ofGesture Features

A.

Task 1

Open-EndedGesture Creation

Number of fingers/strokes, stroke order aredistinguishable than speed or size

SummaryCreating new gestures is hard for end-users• Tendency to focus on the familiar• Opaque nature of gesture recognizer

�

Objective features are more distinguishable• Finger/stroke count, stroke order are

more distinguishable than speed and scale

Quality of gesture sets can be improved• Users are not fully satisfied with their

gesture sets• Low recognition potential

Memorability

Efficiency

Accessibility�

Potential Benefits of Allowing End-User Customization

Take-away Message

Systematic Support is Needed for End-User

Customization

Future WorkMixed-initiative support for customization

Feedback

EditsTrain

System Gesture set User

47

The Challenges and Potential of End-User Gesture Customization

Uran Oh1 and Leah Findlater21 Department of Computer Science2 College of Information StudiesUniversity of Maryland, College Park

�

uranoh@cs.umd.edu | leahkf@umd.edu

Thank you for listening

Questions?