natural and intuitive deformation gestures for one-handed landscape mode interaction · 2019. 5....

Natural and Intuitive Deformation Gestures for One-handed Landscape Mode Interaction

Abstract

The landscape orientation of smartphone offers a better

aspect ratio and extensive view for watching media and

photography. However, it presents challenges of

occlusion, reachability, and frequent re-gripping in one-

handed interactions. To address these issues we took

the opportunity of deformation gestures to interact with

future flexible smartphones. A preliminary survey was

conducted to understand one-handed landscape mode

usage patterns. Then, the 1st study was conducted to

identify 3 most preferred one-handed landscape mode

grips. In the 2nd study, we gathered unique user-

defined deformation gestures to identify the set of most

natural and intuitive gestures corresponding to each

grip. We also found 3 gestures that can be performed

in more than one grip. Finally, we discuss the influence

of the grips on performing gestures.

Author Keywords

Flexible display device; deformation gesture; one-

handed interaction; landscape mode interaction.

CSS Concepts

• Human-centered computing~Human computer

interaction (HCI) • Human-centered

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. TEI '19, March 17–20, 2019, Tempe, AZ, USA © 2019 Copyright is held by the owner/author(s). ACM ISBN 978-1-4503-6196-5/19/03. https://doi.org/10.1145/3294109.3300996

Pranjal Protim Borah

Embedded Interaction Lab,

Indian Institute of Technology

(IIT) Guwahati,

Guwahati, 781039, India

[email protected]

Keyur Sorathia

Embedded Interaction Lab,

Indian Institute of Technology

(IIT) Guwahati,

Guwahati, 781039, India

[email protected]

WIP Session 1 TEI '19, March 17–20, 2019, Tempe, AZ, USA

229

https://doi.org/10.1145/3294109.3300996

computing~User studies • Human-centered

computing~User centered design

Introduction

Spatial properties and arrangements of elements

influence people’s aesthetic responses to an image

irrespective of portrait or landscape aspect ratio [3]. As

an influence of predominant landscape orientation of

Television (TV) screens and computer monitors and for

extensive view with maximum screen real estate [9],

this horizontal orientation (Figure 1a) is often used in

smartphone context for watching videos [10, 9],

recording videos [12], and capturing photographs [12].

However, it presents a variety of challenges, when

interacted with one hand. These challenges include

finger occlusion [6, 15], unreachable target areas [6,

11, 15] on large screen devices and the tendency of

dropping the device due to frequent re-gripping [11, 5].

A few studies have been conducted for comfortable

one-handed landscape mode video browsing [11, 18]

and photo browsing [1]. However, these existing

researches are specific to particular applications and do

not provide a generic solution. Deformation gestures

(Figure 1b) as an input method for flexible handheld

devices present a potential alternative to address the

challenges of one-handed landscape mode interaction.

They have the potential to provide high resolution of

input through eye-free interaction that is independent

of Graphical User Interface (GUI) [17] and provides an

optimal mean for quick continuous bipolar input [2].

Extensive studies on flexible handheld devices were

conducted to identify bimanual deformation gestures

[13, 14, 16] and to address issues of reachability and

occlusion in one-handed portrait mode [6]. Audrey

Girouard et al. had identified a set of deformation

gestures that was easy to perform in a variety of hold

scenarios in one-handed portrait mode [6]. However, to

the best of our knowledge, no studies have investigated

suitable deformation gestures for one-handed

landscape mode usage. In this paper, we present a set

of user-defined deformation gestures for 3 different

user-preferred grips for one-handed usage in a

landscape mode.

Preliminary survey on one-handed use of

landscape mode

The preliminary survey (by means of a questionnaire)

was targeted to build an understanding of landscape

mode usage, handedness of using a smartphone in

one-handed landscape mode, and difficulties during

one-handed interaction with the applications. 50

university students (34 male and 16 female) responded

with a completed questionnaire, between the ages of

18-32 years (Mean = 21.7, SD = 3.4). 44 (88%) right-

handed and 6 (12%) left-handed users participated in

the survey. All the participants had more than 3 years

of experience in using smartphones.

Findings of the survey

Users prefer portrait orientation but often tend to adopt

landscape orientation for better representation of the

content of interest considering the adaptability of the

application. 52% of the participants indicated their

consent in often performing one-handed interaction in

landscape mode. Rest is equally divided into seldom

use (24%) and interacting with both hands (24%).

Majority of the participants (84.2%), who often perform

one-handed interaction preferred to use their right

hand, 13.2% of the participants preferred to use their

left hand and the rest (2.6%) had no preference.

Figure 1: (a) One-handed

landscape mode grip (b)

deformation gesture (Top-right

corner bend out) in one-handed

landscape mode.


230

The most preferred set of applications in landscape

mode include video players, camera applications,

games, and document viewers. We observed similar

challenges presented in literature including unreachable

touch target (81.6%), difficulties in re-gripping the

device during interaction (78.9%), interacting finger

occlusion (47.4%) and difficulty to perform multi-touch

input gesture (34.2%). We also observed that one-

handed landscape mode interaction is mostly preferred

in stationary situations (sitting or standing). In

addition, smartphones possessed by the participants

include 5″ (40%), 5.5″ (46%), 6″ (8%), and other

(6%) display sizes.

Study on User-Preferred Grip

The goal of this study was to identify the set of most

preferred one-handed grips for landscape mode

interaction. This study focuses on the right-handed

users due to the higher proportion of right-handed

landscape mode users (84.2% in the preliminary

survey) and the right-handedness in the global

population [8].

Task The task required participants to demonstrate two most

preferred grips to hold a rigid smartphone with the

right hand in the landscape mode. The grips required to

be the most natural, comfortable and safe (reluctant to

the tendency of dropping the device) grip with

minimum finger occlusion.

Participants 33 university students (22 male and 11 female),

between the ages of 18-30 (Mean = 22.5, SD = 3.2),

participated in the study. All the participants were

right-handed with more than 3 years of experience in

using a smartphone. We made a verbal announcement

in the classrooms to invite students to register their

participation in the study. No remuneration was

provided to the participants.

Apparatus The study was conducted using a “Lenovo K6 Power”

smartphone (5″ display size). The use of rigid

smartphone instead of a deformable prototype was to

avoid artificial grips [5] as users were not familiar with

the use of flexible smartphones.

Procedure

We performed a controlled experiment where each

session was video recorded with the participants’

consent. After the participants read the description of

the task, moderator verbally explained the same to the

participants. To ensure the demonstrated grips to be

natural and comfortable, participants were asked to

imagine and pretend using preferred applications for

which they had experienced the use of one-handed

landscape mode interaction for an extended period of

time (without dropping the device). They were also

asked to speak up the names of the applications and to

mimic similar inclination of the display for comfortable

view without occlusion. No content was displayed on

the screen to avoid any adverse effect of static content

(sometimes even irrelevant) on users’ imagination. The

participants were asked to keep one finger free so that

they can pretend to perform touch interaction on the

screen without any tendency of dropping the device.

This stands the safety of the grip and the notion that

future flexible devices will also continue to support the

traditional touch input method. Participants were asked

not to rest their right hand on an object (table or chair)

or any body part. This particular choice was made due

Figure 2: The most preferred

right-handed landscape mode

grips with at least one finger

available for interaction.


231

to its potential applicability to both sitting and standing

position. Finally, we noted the top two user preferences

for further analysis. All participants completed the task

within 6 minutes.

Results

A total of 57 grips were recorded. It includes 33 grips

as the first preference and 24 grips as the second

preference. As shown in Figure 2 and Table 1, four

unique grips were identified based on the location of

contact between the smartphone and the fingers and

the purpose (support, balance, and interact) of the

involved fingers. All the participants (100%) performed

Grip-A as the first preferred grip while 30%, 28%, and

15% participants performed Grip-B, Grip-C, and Grip-D

respectively as the second preferred grip (Figure 3). We

selected the 3 most preferred grips for our next study.

Grip Support Balance Interact

A Middle, Ring

and Little finger Index finger Thumb

B Thumb Middle and/or

Ring finger

Index

finger

C Index, Middle, Ring and Little

finger None Thumb

D Index, Middle, Ring and Little

finger

Thumb (Occasionally)

Thumb

Table 1: Purpose of the fingers involved in the grips

Study on User-defined Deformation Gesture

This study was aimed to identify the set of most natural

and intuitive deformation gestures with potential to

address the challenges of one-handed landscape mode

interaction.

Task The participants were asked to propose most naturally

recalled and intuitive deformation gestures for each

grip A, B, and C.

Participants

30 university students (20 male, 10 female), between

the ages of 18-32 (Mean = 22.4, SD = 3.6)

participated in the study. All the participants were

right-handed with no prior experience of using a flexible

handheld device. We made a verbal announcement in

the classrooms to invite students to register for their

participation in the study. There were no common

participants from the previous study. No remuneration

was provided for their participation.

Apparatus The study was conducted using a mockup prototype

(Figure 1b) made of flexible PVC sheet (1.5mm

thickness) sandwiched between two foam rubber sheets

(3mm thickness). Deformability introduces challenges

to the development of the mockup in terms of weight

and flexibility ratio. The dimension (14.2cm x 7.2cm x

0.8 cm) of the prototype was selected to resemble the

dimension of a state of art smartphone of display size

5″ (with bezel) and 5.5″ (thin bezel). However, the

weight resemblance was not incorporated to retain the

flexible property of the mockup. We were more

concerned about the flexibility of the prototype than its

weight for better agreement of the gestures [14].

Procedure

We conducted a controlled study where each session

was video recorded with the consent of participants for

further analysis. After the participants read the

description of the task, the moderator demonstrated

Figure 3: Percentage of users

performing the grips.

0 25 50 75 100

Grip-D

Grip-C

Grip-B

Grip-A

1st preference

2nd preference


232

the selected grips and two-handed deformation gesture

(folding the display vertically) using the mockup. We

devoted a minimum of 10 minutes to each participant

to experience the grips and interact with the mockup

until they were ready to start the study. All participants

completed the task within 20 minutes. To ensure

intuitiveness (relatively effortless [4] with reduced

cognitive load and deliberation [7]), participants were

asked to propose referent independent gestures. In

addition, the locations of deformation need to be easily

reachable without switching the grip or compromising

the safety of the grip. The participants were also asked

to perform the gestures without resting the right hand

on an object or any body part. There was no restriction

on the number of gestures proposed by the

participants. The sequence of the grips presented to the

participants was randomized in order to equalize the

effects of gestures proposed in the previous grip. We

considered the most spontaneously derived [7] and

possibly automatic [4] gesture to be the most intuitive

gesture. Accordingly, intuitiveness scores (maximum 7

and minimum 0) were assigned to the gestures based

on the order in which they were proposed. We defined

naturalness of a gesture as the frequency [7] of

participants proposing the same gesture.

Results

30 participants proposed a total of 367 gestures. All

participants proposed 1 to 7 gestures for each grip

except for one participant (p-13) who was unable to

offer any gesture for Grip-B and another participant (p-

27) proposed 8 gestures for Grip-C. Figure 4 and 5

shows the number of gestures proposed in each grip.

Later, we identified respectively 18, 15, and 20 unique

gestures proposed for Grip-A, Grip-B, and Grip-C.

Unique deformation gestures were identified based on

the location of initiating the deformation (top, bottom,

left, right, corner, side, diagonal- along a diagonal line,

horizontal- along the horizontal axis, and vertical- along

the vertical axis), type of deformation (bend, fold -

when touches the opposite surface, and roll) and

direction of deformation (in – movement of edges/

corners towards the user and out – movement of

edges/corners away from the user). This classification

of deformation gestures based on location and type of

deformation is similar to previous studies [6, 14, 16]

except for the direction of deformation. While proposing

the deformation gestures, most of the users referred

the directions as in/out (sometimes inward/outward).

This inspired us to use the words “in” and “out” (also

used in [2]) to match users’ mental model and to avoid

confusion as reported in the literature [6]. Accordingly,

the names of the deformation gestures also consist of

location, type, and direction of deformation. For

instance, in the gesture name “Top-right corner bend

in”, “top-right corner” is the location; “bend” is the type

of deformation and “in” is the direction of deformation

towards the user. We used the frequency and mean

intuitiveness score of gestures to confirm the final set

of most natural and intuitive user-defined deformation

gestures. The final set contains gestures with higher

frequency (F ≥ 15) and mean intuitiveness score (MI ≥

3). Hence, the final catalogue of gestures contains

respectively 4, 3, and 3 gestures for Grip-A, Grip-B,

and Grip-C (Figure 6). Here “Horizontal bend in” (Grip-

A, B and C), “Top-right corner bend in” (Grip-A and C)

and “Horizontal fold in” (Grip-A and B) gestures were

proposed for multiple grips.

We collected individual intuitiveness scores of the

participants corresponding to the gestures in each grip.

Since the collected data are not normally distributed,

Figure 4: Total gestures proposed

in each grip.

Figure 5: Mean of gestures

proposed in each grip.

131

97

139

0

50

100

150

Grip-A Grip-B Grip-C

0

2

4

6

8

Grip-A Grip-B Grip-C


233

we applied non-parametric tests (Friedman Test and

Wilcoxon Signed Ranks Test) to interpret the influence

of grips on the intuitiveness of gesture location, type,

and direction as shown in Table 2.

Grip

Significantly (p<.05) intuitive

Location Type of

deformation Direction

Corner Side

A Top-right - - in

B - Left - -

C Top-right - Bend in

Table 2: Intuitive location, type, and direction of deformation.

Discussion

Both the user studies produced useful insight about

how participants prefer to hold and interact with the

right hand in landscape mode. Observing the recorded

videos comprehensively, we noticed that participants

were familiar with all the grips but switching between

Grip-A and Grip-C seemed to be more effortless. We

found “Horizontal bend in” gesture to be the most

naturally occurring and intuitive gesture that can be

performed in all the three grips. “Top-right corner bend

in” and “Horizontal fold in” are relatively more natural

and intuitive to perform in more than one grip. These

gestures can be mapped to referents of equal priority in

respective grips. In Grip-A and C (Table 2), location

“Top-right corner” and direction “in” are more intuitive.

Similar findings were also reported in portrait mode in

[6]. In Grip-B, location “Left side” is more intuitive as

right corners are primarily involved in gripping the

device. In Grip-C, type of deformation “bend” is more

intuitive as holding the mockup on the palm of the

hand restricted the participants to perform fold gesture.

The range of distinct gestures proposed in each grip

was influenced by the control over the grip during

deformation. Control over the grip during interaction

can be described by the number of fingers involved in

support and balance (Table 1). Maximum control was

observed in Grip-C followed by Grip-A and then Grip-B

during deformation. This allowed users to propose

maximum (20 unique gestures) gestures in Grip-C with

comparatively richer naturalness and intuitiveness.

Conclusion

In this paper, we present a set of user-defined

deformation gestures for right-handed landscape mode

interaction. We conducted the 1st study to identify 3

most preferred right-handed landscape mode grips. In

the 2nd study, we gathered a set of user-defined

deformation gestures corresponding to the 3 grips.

Finally, we identified respectively 4, 3, and 3 unique

deformation gestures for Grip-A, B and C based on their

naturalness and intuitiveness scores. We also identified

that “Horizontal bend in”, “Top-right corner bend in”

and “Horizontal fold in” gestures can be performed in

more than one grip. We further discussed the effect of

the grips on the intuitiveness of performing

deformation gestures. We believe that this study will

provide valuable implications and reference in the

selection of deformation gestures for flexible handheld

devices. As succeeding work, we are investigating on

gesture-referent mappings for one-handed landscape

mode interactions considering the impact of occlusion.

References [1] Sung-Jung Cho, Roderick Murray-Smith, Yeun-Bae

Kim. 2007. Multi-context photo browsing on mobile devices based on tilt dynamics. In Proceedings of the 9th international conference on Human computer interaction with mobile devices

Figure 6: Final set of user-defined

gestures in (a) Grip-A, (b) Grip-B,

and (c) Grip-C (F represents

frequency and MI represents

mean intuitiveness score)


234

and services, 190-197. https://doi.org/10.1145/1377999.1378006

[2] Teemu T. Ahmaniemi, Johan Kildal, Merja Haveri. 2014. What is a device bend gesture really good for?. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 3503-3512. https://doi.org/10.1145/2556288.2557306

[3] Rudolf Arnheim. 1983. The power of the center: A study of composition in the visual arts. Univ of California Press.

[4] Nick Byrd. 2014. Intuitive and Reflective Responses in Philosophy.

[5] Matthew Ernst, Audrey Girouard. 2016. Bending

blindly: exploring bend gestures for the blind. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, 2088-2096. https://doi.org/10.1145/2851581.2892303

[6] Audrey Girouard, Jessica Lo, Md Riyadh, Farshad Daliri, Alexander Keith Eady, Jerome Pasquero. 2015. One-handed bend interactions with

deformable smartphones. In Proceedings of the 33rd annual ACM conference on human factors in computing systems, 1509-1518. https://doi.org/10.1145/2702123.2702513

[7] Sukeshini A. Grandhi, Gina Joue, Irene Mittelberg. 2011. Understanding naturalness and intuitiveness in gesture production: insights for touchless gestural interfaces. In Proceedings of the SIGCHI

Conference on Human Factors in Computing Systems, 821-824. https://doi.org/10.1145/1978942.1979061

[8] Curtis Hardyck, Lewis F. Petrinovich. 1977. Left-handedness. Psychological Bulletin, 84(3), 385-404. http://dx.doi.org/10.1037/0033-2909.84.3.385

[9] Jochen Huber, Jürgen Steimle, Max Mühlhäuser. 2010. Toward more efficient user interfaces for

mobile video browsing: an in-depth exploration of

the design space. In Proceedings of the 18th ACM international conference on Multimedia, 341-350. https://doi.org/10.1145/1873951.1873999

[10] Marco A. Hudelist, Klaus Schoeffmann, Laszlo Boeszoermenyi. 2013. Mobile video browsing with the thumbbrowser. In Proceedings of the 21st ACM international conference on Multimedia, 405-406. https://doi.org/10.1145/2502081.2502242

[11] Wolfgang Hürst, Philipp Merkle. 2008. One-handed

mobile video browsing. In Proceedings of the 1st international conference on Designing interactive user experiences for TV and video, 169-178. https://doi.org/10.1145/1453805.1453839

[12] Yuichiro Kota, Minoru Ikeda. 2006. Mobile phone and camera, U.S. Patent 7,003,318, Filed October 14, 2004, issued February 21, 2006.

[13] Byron Lahey, Audrey Girouard, Winslow Burleson, Roel Vertegaal. 2011. PaperPhone: understanding the use of bend gestures in mobile devices with

flexible electronic paper displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1303-1312. https://doi.org/10.1145/1978942.1979136

[14] Sang-Su Lee, Sohyun Kim, Bopil Jin, Eunji Choi, Boa Kim, Xu Jia, Daeeop Kim, Kun Pyo Lee. 2010. How users manipulate deformable displays as input devices. In Proceedings of the SIGCHI

Conference on Human Factors in Computing Systems, 1647-1656. https://doi.org/10.1145/1753326.1753572

[15] Keyur Sorathia, Aditi Singh, Mayank Chhabra. 2017. BendSwipe: One Handed Target Zooming for Flexible Handheld Display. In IFIP Conference on Human-Computer Interaction, 431-435. https://doi.org/10.1007/978-3-319-68059-0_45

[16] Kristen Warren, Jessica Lo, Vaibhav Vadgama,

Audrey Girouard. 2013. Bending the rules: bend gesture classification for flexible displays. In Proceedings of the SIGCHI Conference on Human


235

Factors in Computing Systems, 607-610. https://doi.org/10.1145/2470654.2470740

[17] Doug Wightman, Tim Ginn, Roel Vertegaal. 2011. Bendflip: examining input techniques for electronic book readers with flexible form factors. In IFIP Conference on Human-Computer Interaction, 117-133. https://doi.org/10.1007/978-3-642-23765-2_9

[18] Yue Wu, Tao Mei, Nenghai Yu, Shipeng Li. 2012. Accelerometer-based single-handed video browsing on mobile devices: design and user studies. In Proceedings of the 4th International Conference on Internet Multimedia Computing and

Service, 157-160. https://doi.org/10.1145/2382336.2382381


236

natural and intuitive deformation gestures for one-handed landscape mode interaction · 2019. 5....

Documents