ux fellows gesture research | gotomedia | sxsw | ieee

#IEEEGoto

Kelly Goto@go2girl #IEEEGoto

Thumbs Up to Natural Gesture Interfaces? Can We Create a Global Gesture Language?

Tuesday, August 12, 14

#IEEEGotoGestures their Origins and DistributionsMorris, Collette, Marsh, O’Shaughnessypregnancyandbaby.comparamount.com


#IEEEGoto


#IEEEGoto

Gestuno » International Sign Language

http://www.handspeak.com

Global Gestures

A native language Gestuno is not. But it is a barrier breaking vocabulary that allows a select few to bridge an international language gap... the need to bridge that international barrier has aroused a sort of lingua franca rooted deep in the trade industry of modern Europe.

http://www.deaflibrary.org

“Government” in different cultures:


http://www.handspeak.com/study/library/index.php?byte=i&ID=50




#IEEEGoto

Global Gestures

Unlike human-to-human gesturing, human-machine gesturing does not have a foundation of thousands of years of history to serve as a context.

- Silicon Labs


#IEEEGoto

Freehand Gestures Are Entering Our Lives


#IEEEGoto

9,000

8,000

7,000

6,000

5,000

4,000

3,000

2,000

1,000

$Million

2012 2013 2014 2015 2016 2017 2018

Gesture Recogni<on Touchless Sensing

Global Touchless Sensing & Gesture Recognition Market Revenue, $million, (2013-2018)

• Expected to reach $15.02 billion by 2018.

• Market value in 2012 was approximately $2 billion.

• Consumer electronics applica<on contributes to more than 99% of the global gesture recogni<on market.

Global Market Revenue Projections


#IEEEGoto

Global Market Revenue Projections

#IEEEGoto


#IEEEGoto

Pointing Gestures


#IEEEGoto

The UX Fellows Gesture Study

#IEEEGoto#IEEEGoto


#IEEEGoto

We Compared Gestures in 18 Countries.


#IEEEGoto

The UX Fellows Gesture Study

Which natural gestures would people from various cultures spontaneously use to control consumer electronic devices?

• Country differences?

• Country clusters?

• Underlying metaphors?

• Acceptance?


#IEEEGoto

Setting 15 min

Warm-up: everyday gestures

#IEEEGoto


#IEEEGoto

Warmup: Everyday ‘Natural’ Gestures

C: I like!

B: I guess I ate something wrong

A: I think I need to see a dentist

What are these people trying to tell us?

D: That meal is really tasty

D: That meal is really tasty


#IEEEGoto

Examined Commands

Switch on device

Volume up

Channel up

Context info for current program

Channel down

Share current program

Volume down

Mute volume

Pause current program

Homepage

EPG

P+

P-‐

Backward navigation in menus

Confirm on-screen dialog (OK)

Deny on-screen dialog (cancel)

Pause video-on-demand (VOD)

Restart VOD

Fast forward VOD

Rewind VOD

Skip chapter VOD

Previous chapter VOD

Stop VOD

Switch off device


#IEEEGoto

Participant Sample

Persons who are able to imagine a gesture-control scenario for a TV.

• Regular usage of a recent HD flat TV (max. 5-6 years of age) and other CE devices and services

• High interest in consumer electronics

• Regular usage of a touch device

• No Xbox Kinect and gesture-enabled TV users

• Potential buyers of international "premium" CE

brands

(no national, retailer or "no name" brands)


#IEEEGoto

Participant Sample

sample

soci

o-ec

onom

icba

ckgr

ound

country


#IEEEGoto

Participant Sample

51%49%

18 countries x 20 subjects = 360 participants

16

65 years

32

min

max

mean


#IEEEGoto

The Results

#IEEEGoto#IEEEGoto


#IEEEGoto

Difficulty of TV-Related Gestures

Self-‐reported difficulty, 5 = very difficult, 1 = very easy, n = 360 tech-‐savvy CE consumers in 18 countries

easy

diff

icul

t


#IEEEGoto

Commonality of Gestures

no. of countries with the same main gesture

High Medium Low


#IEEEGoto

Commands with High Commonality

Volume down

Volume up

Pause VOD


Rewind VOD

Fast forward VOD



#IEEEGoto


Volume down

Volume up

Pause VOD


Rewind VOD

Fast forward VOD


Commonality 18 of 18

Difficulty (1-‐5) 1.8

Q: How do you turn down the volume?


#IEEEGoto


Volume down

Volume up

Pause VOD


Rewind VOD

Fast forward VOD




Q: How do you turn up the volume?


#IEEEGoto


Volume down

Volume up

Pause VOD


Rewind VOD

Fast forward VOD




Q: How do you pause viewing a film?

NL


#IEEEGoto


Volume down

Volume up

Pause VOD


Rewind VOD

Fast forward VOD




Q: How would you pause the current TV program?

NL


#IEEEGoto


Volume down

Volume up

Pause VOD


Rewind VOD

Fast forward VOD




Q: How do you rewind? (with an image being shown)

AU, UK

NL


#IEEEGoto


Volume down

Volume up

Pause VOD


Rewind VOD

Fast forward VOD




Q: You are watching a boring scene, how do you fast-‐forward (with an image being shown)?

AU, UK

NL


#IEEEGoto


Volume down

Volume up

Pause VOD


Rewind VOD

Fast forward VOD




Q:: You have chosen a film from an online video library. The TV asks you "do you really want to buy the film?" How do you answer "yes"?

CN, IT, KR


#IEEEGoto

Commands with Medium Commonality

Channel up

Channel down

Stop VOD

Mute volume

Backward navigation in menus

Deny on-screen dialog (cancel)

Skip chapter VOD


#IEEEGoto


Channel up

Channel down

Stop VOD

Mute volume

Backwards navigation in menus

Deny on-screen dialog

(cancel)

Skip chapter VOD



Q: You are zapping through the TV programs: How do you change from channel 15 to channel 16 for example?

P+

AU, ES, KR, US

CA


#IEEEGoto


Channel up

Channel down

Stop VOD

Mute volume



(cancel)

Skip chapter VOD



Q: You are zapping through the TV programs: How do you change from channel 16 to channel 15 for example?

AU, ES, KR, US

CA

P-‐


#IEEEGoto


Channel up

Channel down

Stop VOD

Mute volume



(cancel)

Skip chapter VOD



Q: How would you stop watching the film altogether (because you want to do something different)?

CN, KR

FI, RU

NL


#IEEEGoto


Channel up

Channel down

Stop VOD

Mute volume



(cancel)

Skip chapter VOD



Q: How do you set the volume to "mute"?

AU, CN, MX FI, RU

BE

UK


#IEEEGoto


Channel up

Channel down

Stop VOD

Mute volume



(cancel)

Skip chapter VOD



Q: Imagine that you were looking for a film in an online video library. You have gone to the comedies sec<on and have chosen the descrip<on of a single movie. Now you want to return to the previous set of op<ons (all of the comedies). Which gesture would you use for that?

AU, BE, CA

AR, RU

ES, MX, US


#IEEEGoto


Channel up

Channel down

Stop VOD

Mute volume



(cancel)

Skip chapter VOD

Commonality 9 of 18


Q: How do you express that you don’t want to buy it?

AU, CN, RU, TR

KR

AR, BE, IT, MX


#IEEEGoto


Channel up

Channel down

Stop VOD

Mute volume



(cancel)

Skip chapter VOD

Commonality 9 of 18


Q: How would you skip to the next chapter of the film?

BE, CN, ES, FI, FR, IN, MX, UK

NL


#IEEEGoto

Commands with Low Commonality

Switch on device

Switch off device


Restart VOD


EPGContext info for currentprogramHomepage


#IEEEGoto


Commonality 8 of 18


Q: How would you switch on the TV?

Switch on device

Switch off device


Restart VOD



BE, FR, US

KR

DE, IT

TR

CNCA

FI


#IEEEGoto


Commonality 7 of 18


Q: How would you switch off the TV? Note: No common gesture in Italy

Switch on device

Switch off device


Restart VOD



ES, MX, NLIN, KR

RU

TR

CA

AR


#IEEEGoto


Commonality 8 of 18


Q: How would you return to the previous chapter of the film?

Switch on device

Switch off device


Restart VOD



BE, CN, ES, FI, FR, MX, NL

IN, US

UK


#IEEEGoto


Commonality 7 of 18


Q: How do you start it (film) again? Note: No common gesture in UK

Switch on device

Switch off device


Restart VOD



AU, ES, KR, MX

BE

NL RU

US

AR, DE


#IEEEGoto


Commonality 6 of 18


Q: How would you share with your friends what you are currently watching or listening to? Note: No common gesture in UK

Switch on device

Switch off device


Restart VOD



CN, FR, IN

FI

AR, TR

MX

NLKR

RU

US


#IEEEGoto


Commonality 5 of 18


Q: You'd like to see an overview of tonight's TV programs of the main channels (EPG). How do you ac<vate this func<on with a hand gesture? Note: No common gesture in Italy

Switch on device

Switch off device


Restart VOD



AU, US CN, FI, IN, KR, RU

BE, UK

TR

NL

MX


#IEEEGoto


Commonality 4 of 18


Q: How would you get more informa<on for the running TV program?

Switch on device

Switch off device


Restart VOD



FR, KR, RUAU, FI

CN, IN, TR

AR

DE

BE

ES

US

UK


#IEEEGoto


Commonality 4 of 18


Q: Now you want to go to the ini<al set of op<ons -‐ that means the first op<ons you see when you switch on the device.Note: No common gesture in Italy

Switch on device

Switch off device


Restart VOD



AR, AU, MX

DENL, UK

FI

US

BE, IN

CA, ES, TR


#IEEEGoto

Some General Insights

#IEEEGoto


#IEEEGoto

Direct vs Abstract Gestures

We observed two kinds of natural gestures.

● "Direct" gestures

¡ strongly influenced by touch devices

● Abstract gestures

¡ basis in language, iconography and metaphors

¡ associated with more complex tasks

¡ lower intercultural commonality


#IEEEGoto

1.0

1.5

2.0

2.5

3.0

TR FR MX AR IN FI CN RU NL DE AU US BE ES CA IT KR UK

Everyday gestures CE gestures

difficulty

min

max 5.0

Average Gesture Difficulty by Country

…


#IEEEGoto

Average Gesture Difficulty by Country

1.0

1.5

2.0

2.5

3.0

FR CN TR IN AR RU MX US IT DE AU FI CA BE ES NL UK KR

CE gestures

…

difficulty

min

max 5.0


#IEEEGoto

General Acceptance

§ Our 360 tech-savvy participants responded very positively to the idea of semantic gesture input for TVs. "fun to use", "convenient", "coolness factor”

§ Participants would like to use gestures to control other devices as well (e.g. audio).


#IEEEGoto

General Acceptance

Can CE gestures be deducted from everyday gestures?

USAInfluenced by touch screens — most thought gestures should be customizable.

CanadaPersonal hobbies eg: gestures inspired by skiing, kick-boxing etc. Sign language

Russia.Not a lot of everyday gestures used. Replicating alphabets, creating new gestures and pointing used.

No.

Yes.Mexico Smart phones and touch screen devices eg: sliding the menu up from the bottom, swiping gestures etc.

Spain | Mexico | France | NetherlandsEveryday gestures and touch screen gestures

Australia | Germany | ItalyEveryday gestures

Argentina | China | KoreaTouch screen devices

IndiaAnalog gestures, touch screen gestures and icon based gestures

BegiumEveryday gestures, gestures while using electronic devices and touch screen gestures

FinlandEveryday gestures and touch screen gestures


#IEEEGoto

Private vs Public Display

§ Considerations

§ Ergonomics

§ “Gorilla Arm”§ Recognition

privatepublic


#IEEEGoto

Operation

Many participants felt that voice and sound could be a good support for gesture input.

waking up the device with a sound

Can the TV hear me?

alterna4ve op4on based on context and need


#IEEEGoto

Operation

FR CN TR IN AR RU MX US IT DE AU FI CA BE ES NL UK KR

Seman<c

Poin<ng

Many users would prefer switching between gesture types.

FR CN TR IN AR RU MX US IT DE AU FI CA BE ES NL UK KR Σ

Seman<c l l l l l l l l l l l l l l l 15

Poin<ng

FR CN TR IN AR RU MX US IT DE AU FI CA BE ES NL UK KR Σ

Seman<c l l l l l l l l l l l l l l l 15

Poin<ng l l l l l l l l l 9

associated with more advanced commands alterna4ve op4on based on context and need


#IEEEGoto

Typical Concerns of Users

§ Learning and retention of gestures

(but: would be easy if based on implicit

behavior repertory)

§ Sensitivity and accuracy of devices

§ Interference by other gestures in the

room

§ Compatibility with lean-back disposition


#IEEEGoto

Conclusions

1. For basic CE commands a common language for natural gestures already exists across cultures.

2. More complex commands could be realized by either

a. customizing these gestures for each individual market

b. customizing individually

c. proposing a set of semantic gestures that need to be learned (the study provides some suggestions)

d. switching to pointing gestures in combination with a GUI

3. It is necessary to consider cultural differences at a country level


#IEEEGoto

Future of communication

Do we want to create a better remote control, or do we want to create a 2-way language of communication?

- go2girl


#IEEEGoto


LG Magic Remote


#IEEEGoto

The problem: no standard gestures currently exist.

Many new devices in the world of ‘touchless tech’ are using completely different gestures for the same common functions.



#IEEEGoto

Can we utilize this methodology to start a conversation and bring the deaf community, the large technology giants and standards together to develop a universal gesture language?

source: autocarhire.com



#IEEEGoto



#IEEEGoto

The governance crisis point

Quantity & complexity of information

Ability to deal with quantity & complexity

of information

Zap Think LLC

TIME

(innovation) COMPLEXITY vs. COPING (desire)


#IEEEGoto

Remember: People don’t advance as quickly as technology.

BEYOND USABLE


#IEEEGoto

CONTEXT


#IEEEGotosource: wikipedia

the future?


#IEEEGoto

Find all results on www.uxfellows.com

Drop us a line: [email protected]

Michael Woermann [email protected]

www.3bridgesaustin.com


http://www.uxfellows.com/

http://www.uxfellows.com/

mailto:[email protected]




http://www.3bridgesaustin.com

http://www.3bridgesaustin.com

#IEEEGoto

● UX design + research consulting

● San Francisco, CA

● Global mobile focus

#IEEEGoto


ux fellows gesture research | gotomedia | sxsw | ieee

Technology

ieeegoto global gestures

ieeegoto thumbs

ieeegoto warmup

ieeegoto freehand gestures

ieeegoto kelly goto

everyday gestures

global gesture language

everyday natural gestures