1571.pdf

8/19/2019 1571.pdf

http://slidepdf.com/reader/full/1571pdf 1/5

Using eye tracking to apply focus detection to an adaptive

game controller

Blind for review

ABSTRACT

In the gaming experience, players are used to intuitive interfaces

that allow them to jump straight to the entertainment. Modern joysticks are physical hardware components with a fixed layout,

being the main interface for a large variety of games with different

control methods and needs that will be played by users that also

present different ergonomic needs and preferences. This work

proposes a new interface, based on a touchscreen device. We

present a gamepad concept capable of adapting itself dynamically

to a user according to its touch and attention focus patterns, trying

to avoid errors and provide a more comfortable experience. We

also present the results of our usability tests, with both objective

and subjective evaluations and the discussion about our findings.

CCS Concepts

• Human-centered computing

Human computer interaction

(HCI)

Interaction devices

Touch screens • Human-centered computing

Ubiquitous and mobile

computing Ubiquitous and mobile devices

Mobile

devices

Keywords

Adaptive interfaces; adaptive game control; game input; eye

tracking; focus detection.

1. INTRODUCTIONVideo games are one of the main entertainment fields nowadays.

They are composed of many elements, such as gameplay, audio,

graphics and narrative. When these factors are well performed and

combined, the game may produce engagement and immersion to

the players. Nowadays a special attention has being given to the

game interaction, which is an important element for the

implementation of the gameplay [7] [3] and rules [11]. Due to this

fact, the controller and the control scheme must match userexpectations or even exceed them [12]. To achieve and enhance

the game interaction, new ways of inputs and devices has been

proposed [ANONYMOUS].

These inputs must be intuitive, provide a deep interaction, where

sometimes the user can map the action in the real world to the

game. A successfully control scheme must improve the gaming

experience. On the other hand, a game that does not have an

intuitive control and makes the user uneasy when playing may

decrease the overall entertainment process.

To provide a better user experience, we proposed in past works

the use of an adaptive controller using a mobile device to create a

dynamic touchscreen interface. Our controller can present any

layout, allowing game designer to project not only their games, but the own joystick used to play it. Currently, games are played

using physical joysticks that present a generic button

configuration and layout, with a fixed amount and position for its

buttons. Our work aims to remove this constraint, allowing each

game to use a controller layout that best fits its needs. In

following works we proposed the dynamic improvement of the

custom interface based on machine learning techniques and data

captured from the user’s touches on the screen. In this work we

improve our proposal including attention focus elements based on

eye tracking approaches in order to best fit the ergonomic needs

and the current moment in the game.

2. RELATED WORKThis work presents the use of a mobile device as an adaptive

controller to games. Actually there are many different kinds of

controllers. To name few of them, we can mention gamepads,

keyboards, steering wheel and mobile devices. The study in [4]

compared the usability, user experience, functionality and the

design of some controllers but they did not compare mobile

devices as inputs neither adaptive devices. Mobile phones have

specific hardware (camera, accelerometer, GPS, Bluetooth and so

on) with lots of them being different from the ones found in

traditional game platforms, like video games and PCs. For this

Permission to make digital or hard copies of all or part of this work for

personal or classroom use is granted without fee provided that copies are

not made or distributed for profit or commercial advantage and that

copies bear this notice and the full citation on the first page. To copy

otherwise, or republish, to post on servers or to redistribute to lists,

requires prior specific permission and/or a fee.

SAC’16 , April 4-8, 2016, Pisa, Italy.

Copyright 2016 ACM 978-1-4503-3739-7/16/04…$15.00.

http://dx.doi.org/xx.xxxx/xxxxxxx.xxxxxxx

8/19/2019 1571.pdf


reason, these devices bring new forms of user interaction,

however with the drawback of lacking tactile feedback [6].

Adaptive user interfaces are interactive software systems that

improve its ability to interact with a user based on partial

experience with that user [8]. Some of them will be detailed in

this section to provide a useful background for this work.

Mobile phone touch screen devices are very common. Most

devices that have such features possess few buttons, with almost

all input made by touch. Hence, mobile touchscreen games must

be designed to accept most of their input by touch that will be

used in games in a similar way to mouse clicks on a regular

computer allowing developers to make different types of mobile

games based on virtual buttons. Consequently, the screen can

draw buttons and use it to simulate button input.

In previous work [ANONYMOUS] the authors developed a

controller that adapt itself according to the users touch press. The

statistical analysis showed strong evidences that the adaptive

controller improved the accuracy of the users.

This work introduces new heuristics for controller adaptation

based on eye tracking that recognizes where the user is looking

and then decides if the adaptation is necessary. The use of eyetracking to estimate user’s gaze location is present in several

works, such as [9]. It also has being used on games, mainly as a

control method, a technique used on [14] and to evaluate the

user’s focus in gaming interfaces [15]. In our case, we will track

the user’s focus to dynamically evaluate the best parameters for

our adaptive interface.

Figure 1: A user playing a game with the adaptive controller.

3. PROPOSED INTERFACE

Our adaptive controller consists in a mobile application for asmart device (in our case, an Android smartphone or tablet) that

presents a customizable graphic interface, specifically built to

attend the game’s needs. The contr ols are presented in the screen

and the user interacts with it by touching buttons to perform

actions in a game running on a regular PC or game console.

Figure 1 shows the prototype controller in action.

However, the interaction with the controller is not a static

experience. Each user has personal ergonomic needs and

preferences, so that a generic controller is not capable of

providing the best experience for the individual. Besides that, the

game itself is not constant: the challenges, level design and even

control options will change as the player progresses on it. To

create a dynamic interface that follows this process, our adaptive

controller constantly improves its interface to better fit both the

player and the current moment in the game. Adaptations may be

triggered by different adaptation causes, such as the context of the

interaction, the experience of the user, or user behavior [2]. Theuse of a touchscreen device instead of a physical hardware, like

traditional controller, allows a single device to not only provide

custom interfaces for each game, but to change its own layout so

that it can respond to new requirements in the interaction.

In [ANONYMOUS], the authors developed agents that monitor

the player usage during the gameplay experience, tracking the

user’s touches and buttons interactions. Based on this data and

using machine learning approaches, the system dynamically

changed and adapted the buttons, in order to better fit the specific

player. In this work we propose a novel approach for the adaption

using eye tracking to perform better adaptations. In our

experiment we will focus in changes to the button’s position and

size, but other changes such as shape and even the way to interact

with a button can be altered, since our controller allows any kindof interface element, including dragging into the sensible screen.

The communication between the mobile app on the smart device

and the PC that is running the game is performed via network,

using the traditional TCP protocol. Both devices must be in the

same network and an application on the device that is running the

game (PC or console) will be responsible for receiving the

commands from the controller and translate it to local keyboard

events that perform actions on the game. In this work we also

capture eye tracking data and send it to the controller.

4. CONTROLLER ADAPTATIONS BASED

ON THE USER BEHAVIOR

4.1

Adaptation Based on the User’s Touches All user touches on the screen are tracked and kept in an internal

database, which will use the 10 most recent touched points and

the amount of presses on each button for evaluating and perform

adaptations to improve the user experience. For our tests, we

manipulated buttons size and positions.

The size adaptation is based on the following heuristic: The

amount of touches is stored for each button and the controller

creates an ordered list, where the first button is the one that was

pressed more times while the last one will be the least used

button. One third of the most used buttons will have their size

increased while one third of the less used buttons will decrease in

size. We limited in our tests the button size increase to a

maximum of two times its original size, while the lower bound

was established to its own initial size. This list is constantlyupdated, so if a button stops being used frequently, it will

naturally move down the list and decrease in size.

For the position adaptation, it was used the K-means clustering

algorithm [13], an unsupervised machine learning method. The K-

means receives a set of points and try to cluster them in K classes,

each one containing the corresponding points and a centroid. This

centroid will be the most important result, since it represents the

center of all touches in a class. During the interaction, the user

performs correct touches in the buttons and incorrect touches in

8/19/2019 1571.pdf


the surrounding areas. With K-means, these touches will be

grouped and the centroid will represent a position that allows the

button to better represent the area where the user is actually trying

to press. Each button will be paired with the closest centroid

found (if a centroid is close to it) and its position will be changed

so that the button’s center matches the centroid.

All these changes are made gradually, with each button changing

position or size by a maximum of constant defined pixels periteration. The adaptations also follow some rules and constraints.

Each time a button has to change its size or position, the

controller will verify if this change would make the button invade

another button’s area (an intersection). In positive case, the

change will not be made. However, it is important to note that the

button will grow or move as much as it can without intersecting

with a neighbor and that, if future layout changes moves the

neighbor out of the way, the desired change will now continue.

To better analyze the effect that our improved controller can have

on the general user experience, we used an eye tracking algorithm

to collect information about focus during the gameplay

experience. This data is used as another input to adaptation as will

be better explained in the next section.

4.2 Adaptation Based on Focus DetectionGaming aims to be an immersive experience. In our study, we will

consider that the ideal controller should not distract the player

from the game and must avoid breaking this immersion. As our

controller relies on a touchscreen and the lack of physical

feedback can make the user miss buttons, a possible immersion

break can be observed. If the user cannot feel the position of the

button, he will have to look to the controller to determine visually

the button’s position. While the K -means algorithm is used to

optimize the button’s layout and correct the interface to avoid

errors, they still are a guess. With this situation on mind, we used

an eye tracking algorithm presented in [16] and modified it to

determine if and when the user may be losing the focus on the

game and looking back to the controller.

These events will be used to allow the interface to perform a

dynamic self-evaluation. In previous user tests [ANONYMOUS],

the authors observed that the user interface would change

constantly even if the user’s play style was not changing. The

feedback received from the users pointed that sometimes the

adaptation was too aggressive. This resulted in some cases where

an interface already close to the optimal just kept changing,

sometimes resulting in less adequate configurations. Another

interesting observation was that the users would look to the

controller when they couldn’t find the buttons they were trying to

press. As the controller uses a touch interface, the lack of physical

feedback, something always present on regular joysticks, results in

a case where the only possible way to locate a button is to look at

the controller. But as the adaptive interface results in an increased precision, the need to visually check the position for a button can

be reduced when the interface reaches a configuration closer to

the optimal layout for a user.

We decided to track the user’s attention focus to provide a new

source of data for our adaptations. We used the computer’s

webcam to track the pupils of the user and calculate if he is

looking to the screen or to the controller, using [16] method for

eye tracking. Our prototype controller would adjust the

aggressiveness of its adaptation according to the user’s focus. If

the user looks constantly to the controller, the speed of change for

the size and position of the buttons will be progressively

increased. If the interface stabilizes and the user stops looking at

the controller for more than 10 seconds, the controller will slowly

decrease the speed of the adaptation, stabilizing the interface and

performing way less changes to its layout.

We change the speed of the adaptation using two parameters: the

maximum change of a buttons’ size and position per iteration (the

algorithm is executed 2 times per second) and the amount of

points passed to the K-means clusterization algorithm. The first

parameter, when increased, makes the controller apply the

changes faster, changing its layout almost immediately, while a

lower value will result in slower changes. The last parameter,

when decreased, results in less points being passed to the

clusterization algorithm, resulting in an adaptation focused in the

more recent interaction patterns, being able to change its layout

more dramatically to answer to differences in the user’s behavior.

When this parameter is increased, the controller will base its

suggestions in long term characteristics of the gameplay section

and will be more conservative when performing changes.

We expect that this approach helps to avoid cases where the

interface keeps adapting itself after finding an optimal

configuration, performing unnecessary changes that can be

detrimental to the user’s experience.

5. USABILITY TESTSIn order to evaluate properly our proposal adaptation and the

interaction with the final user, we conducted a usability test,

observing some parameters given by the controller and the eye

tracking algorithm. To realize this evaluation, the tests were

divided in two stages: the pilot and the final user tests. The pilot

was the preliminary test and it worked to set the parameters to the

final test, determining the best configuration possible that would

fit to the game and the adaptation. As the controller do not need to

be identical to the physical one, the pilot test gave insights to

define his design.

5.1 Participants and ApparatusThe evaluation tested two different controllers, that apparently

resemble visually but the core algorithm is different. When the

user looks to the screen, the first impression is that both

controllers are equal. But, one controller is adaptive and the other

not. It is important to note that while the user plays, the adaptive

controller changes completely including the visual.

The functionality of both is the same, so each button corresponds

to the same action regardless of version. The participant must test

both controller and he does not know which version is being

tested at the moment.

The group of testers was composed by 8 users, with age that

varies between 21 and 51. Our group consisted of 3 women and 5

men. The whole group tested both controller versions with one

game, Streets of Rage. The user’s level of experience and profile

varied from experienced gamers to casual players and people that

usually do not play games at all, covering a wide range of possible

player profiles. In which play for more than ten years and still

play.

8/19/2019 1571.pdf


8/19/2019 1571.pdf


subjective evaluation have shown that a final analysis requires a

new test that totalizes the amount of gazes targeted to the

controller by the users.

Figure 3: The initial controller layout (upper side) and the

final configuration achieved by the adaptive controller for a

user (lower side)

7.

CONCLUSION AND FUTURE WORKSMany people avoid start playing games. One of the reasons is the

need to interact with complex devices with large amounts of

buttons and combinations, an issue capable of pushing them away

from video games.

In this work we provided an experience where each user can have

its own personalized game controller, automatically adapted to its

ergonomic and personal preferences. Our test results showed that

the adaptive controller can increase the user’s precision, leading

to less errors and a more comfortable interface. Additionally, the

subjective evaluation also demonstrated good acceptability by the

users.

After proposing a new controller that can be adapted to each user

behavior, our intention in a future work is to include new tools in

the proposed work that can be used by the game interface designerto devise his own initial interface with adequate amount and

layout of the buttons. In this new paradigm, both the game and the

machine learning algorithms would work together to personalize

the interface. In this case, we would have a game controller for

each user and for each game, resulting in unique combinations.

Measuring the user experience is also an area that must be

explored. With this in mind, we would like to use an EEG

(electroencephalography) headset to evaluate the user’s emotions

during gameplay, trying to determine the exact impact that a game

controller can have on variables such as frustration, engagement

or excitement.

8. REFERENCES[1] Bangor, A., Kortum, P., and Miller, J. Determining what

individual SUS scores mean: Adding an adjective rating

scale. Journal of usability studies, 4(3), 114-123, 2009.

[2]

Bezold, M., and Minker, W.: Adaptive multimodal

interactive systems. Springer, Boston, 2011.

[3] Brooke, J. SUS-A quick and dirty usability scale. Usability

evaluation in industry, 189(194), 4-7.Chicago, 1996.

[4] Brown, M., Kehoe, A., Kirakowski, J., and Pitt, I. (2010).

Beyond the gamepad: HCI and game controller design and

evaluation. In Evaluating User Experience in Games (pp.

209-219). Springer London.

[5] [ANONYMOUS], details omitted due to double-blind

reviewing.

[6] Joselli,M. ,Silva Junior, J. R. S., Zamith, M., Clua, E., and

Soluri,E. A content adaptation architecture for games. In:

SBGames. SBC, 2012.[7] Koster, R.: Theory of fun for game design. O’Reilly Media

Inc., Sebastopol, 2013.

[8] Langley, P. Machine learning for adaptive user interfaces. In:

Brewka, G.,Habel, C., Nebel, B. (eds.) KI 1997. LNCS, vol.

1303. Springer, Heidelberg, 1997.

[9] Majaranta, P., and Bulling, A. Eye tracking and eye-based

human – computer interaction. In Advances in Physiological

Computing (pp. 39-65). Springer London, 2014.


reviewing.

[11] Salen, K., and Zimmerman, E. Rules of Play: Game Design

Fundamentals. Cambridge, Massachusetts: The MIT Press.

ISBN 978-0-262-24045-1. “Game play is the formalizedinteraction that occurs when players follow the rules of a

game and experience its system through play”, 2004.

[12] Schell, J.: The Art of Game Design: A book of lenses. CRC

Press, Boca Raton, 2008.

[13] Smola, A., and Vishwanathan, S.: Introduction to Machine

Learning. Cambridge University, UK, 3234 (2008)

[14] Sundstedt, V. Gazing at games: using eye tracking to control

virtual characters. ACM SIGGRAPH 2010 Courses. ACM,

2010.

[15] Sundstedt, V., Bernhard, M., Stavrakis, E., Reinhard, E., and

Wimmer, M. Visual attention and gaze behavior in games:

An object-based approach. In Game analytics (pp. 543-583).

Springer London, 2013.

[16] Timm, F., and Barth, E. (2011, March). Accurate Eye Centre

Localization by Means of Gradients. In VISAPP (pp. 125-

130).


reviewing.


reviewing.

1571.pdf

Documents