1
Usability Evaluation Method through
SUS Analysis for Coin Party
UROOJ WAHEED M. SADIQ ALI KHAN HIRA ANWAR KHAN
ASSISTANT PROFESSOR ASSOCIATE PROFESSOR LECTURER
COCIS, PAF-KIET DCS, UoK COCIS, PAF-KIET
Karachi, Pakistan Karachi, Pakistan Karachi, Pakistan
[email protected] [email protected] [email protected]
Abstract
Usability explains the procedure about how a product or system is providing interaction to the user. While user is interacting with an API usability gives a view about the degree of user experience. Usability is about effectiveness, efficiency and the overall satisfaction of the user. The performance and satisfaction metrics are the base line for developing of Usability tasks. For making the methodology to evaluate the system a game name “Coin Party” is chosen. In this paper authors uses different techniques and methods to not just evaluate a design of the API but also factors like; system status visibility, ease of use, control of user, freedom of user, mapping among real world and system.
Keywords: Usability Evaluation Performance Metric, SUS and Self-Reported
Metrics. 1. Introduction
Usability defines the extent to which the system and product is usable to the user,
i.e.; the interaction between system and user. Through usability we can also measure
the degree of experience user achieve while interacting with the system. There are
three major components which describe usability: effectiveness, efficiency and the
overall satisfaction [5]. Evaluation of usability analyzes the grade to which user is
allowed for interaction with the API easily and enjoyably [4]. The design of the
system can be improved by using Usability evaluation methods.
1.1 Testing Software: Game
This paper applies Usability evaluation method on the API of a game named as Coin Party. Coin Party game is coin pushing game based on jungle story. The game is developed by group of Pakistani game developers known as Mind storm studios.
The game features are described below.
International Journal of Pure and Applied MathematicsVolume 118 No. 9 2018, 171-181ISSN: 1311-8080 (printed version); ISSN: 1314-3395 (on-line version)url: http://www.ijpam.euSpecial Issue ijpam.eu
171
2
1.1.1. Complete your Quests: Completing a quest has the ultimate rewards within the game. Each quest will transform an existing table power up in a special way and even unlock new Power Ups.
1.1.2. Compete on the Hall of Fame: This is the ultimate honor in Coin Party. Compete with players across the world to see who can bank the most coins.
1.1.3. Upgrade your Power Ups: When you upgrade your power ups by collecting prize recipes, you end up collecting coins faster and the game play just becomes a lot more fun with wild things happening on the table all the time.
1.1.4. Increase your Level: Level up to access higher level quests and each level gives makes you naturally more powerful with faster coin regeneration, more prizes and more magic chips.
1.1.5. Collect Prizes: Prizes help you unlock quests, upgrade magic chips and you can exchange them for coins and party bucks from the Prizes menu.
1.1.6. Unlock Achievements: Visit the Hall of Fame to see which achievements you can unlock. We will be adding plenty more achievements with new updates.
1.2 Description of Coin Party
When you start the game, it shows that the heaps of gold coins are dropping from
the top onto the dozer which later drops the coins and prizes from the play board. One
can appreciate the entire gambling club involvement with turn haggles machines [1].
To open more catalyst and mysterious chips in the quests, a player will use the earned
prizes and coins in exchange. The players can gather club turn tokens and opening
International Journal of Pure and Applied Mathematics Special Issue
172
3
gold bars that show up on the dozer table and spend them on turning the in-
amusement fortune prize haggle space machine for astounding prizes [8].
2. Method
To evaluate the system we focuses on the design of the system but also additional
features included like ease of use, visibility of game control, user control and freedom
to play game, synchronization between system and the real world. In this study we
have used self-reported metric and on the basis of it we have make our methodology
by looking at SUS Score.
2.1 Evaluation method for Usability
As the demand of the evaluation increases at industrial level, there emerges a need
of evaluation technique which is cost effective and also practically possible to carry
out on large scale. From this technique the system performance can be measured with
respect to predecessors or with competitors.
John Brooke states that: “The measure had to be capable of being administered
quickly and simply, but also had to be reliable enough to be used to make
comparisons of user performance changes from version to version of a software
product” [12]. If the test consists of questionnaire which includes more than 25
questions, firstly this will frustrate the user to complete the test, and if he left it
incomplete then it will be of no use in the evaluation. In return become hassle for
users to perform test. The user gets involved easily in the test which has simple and
less amount of questions and require less time for performing the test. In contrast to
the problems stated above the simple evaluation technique was discovered. This
method provides the global view of subjective assessments of usability.
2.2 System Usability Scale
In 1986 the new system usability evaluation method was created by John Brooke.
From then till now, SUS is considered to be well known rating scale for evaluation of
usability because of its simplicity and cost effectiveness. Through SUS we are able to
gather user’s subjective responses [6]. SUS is pocket friendly i.e. cost effective,
because it consists of questionnaire with 10 questions which requires less resource
with reliable result. SUS is quicker to implement because the template of the
questionnaire is already present you just need to upload it for initiating the work. SUS
gives reliable results even when the sample size is small. It also effectively
categorizes the system on usable and unusable classes. SUS gather the responses of
user in the form of Likert scale. SUS method asks users 10 questions with 5 response
options from strongly disagree to strongly agree.
International Journal of Pure and Applied Mathematics Special Issue
173
4
The SUS questioner consists of following 10 questions:
System Usability Scale
Strongly Disagree
to
Strongly Agree
1 2 3 4 5
1 I think I would like to play this game frequently. 2 I found the game unnecessarily complex.
3 I thought the game was easy to use.
4 I think that I would need the support of a technical
person to be able to use this game.
5 I found that various options in this game were well
integrated.
6 I thought there was too much inconsistency in this
game.
7 I would imagine that most people would learn to use
this game very quickly.
8 I found the game very cumbersome to use. 9 I felt very confident playing the game.
10 I needed to learn a lot of things before I could get going
with this system.
Table 1: Questions for System Usability Scale
2.3 Calculating Usability Score using SUS
SUS gives a single numeric value which represents the combined usability of the
overall system. For calculating the SUS score the response of each question is
transformed to a number keeping in view the number of question [7], i.e. even number
question or odd number question. Let us assume that the response of user on the scale
value will be “X” then for all even number questions we will be using formula: “5-X”
and for all odd number questions we will be using formula “X-1” .
The answer we get from the above mentioned formula will be in the range of 0 –
40 so we will map this range to 0 – 100 by multiplying the SUS value to 2.5. The 0 –
100 a percentile range rather than percentage.
International Journal of Pure and Applied Mathematics Special Issue
174
5
Figure 1: System Usability Scale
3. Procedure Used for Evaluation
3.1 Questioner Development For gathering the feedback of user about game and his experience of
interacting with the system we use System Usability scale Template which
consists of 10 questions mentioned in Table 2 [3]. As the SUS questions have
quantitative responses so will get accurate result. This quantitative result will be
helpful in improving the game [13].
3.2 Methodology
The evaluation tests for usability were conceded from the users’ age group 19
– 25, mostly students of all four years of BSCS. The survey was conducted using
Google Form Template for the SUS questionnaire [2]. The reason for choosing
Google Forms is that the test was conducted for the game on MOBILE phone
which also facilitates the user and lessens the hassle of paper pen work. The
Google Forms can be accessible on any MOBILE phone easily by opening the
link on the browser.
The test sessions were conducted in closed environment of PAF – KIET class
room with the help of volunteers in our supervision where the MOBILE phones
were provided to each participant with the initial briefing about the evaluation
test using SUS before playing the game and filling the survey.
3.3 Participants The samples of users are selected for the surveys are from the age group of
18 – 25. 22 participants are selected in total. The selection for the mentioned
above age group was made keeping in consideration that firstly we need to test a
game and secondly the game which we selected to test was available on android
International Journal of Pure and Applied Mathematics Special Issue
175
6
and iOS i.e. touch screen mobiles/ tablets. So the above mentioned age group
posses both the qualities.
The participants were provided with the android / iOS device. They were
asked to comment on the usability and learn ability of the game. 5 volunteers
were selected for conducting the survey fairly. Firstly the 20 minutes session was
conducted to brief about the game and procedures of the survey to the
volunteers. Then each volunteer is assigned to 1 participant. Only 5 participants
and allowed in one time frame because of limited resources i.e. devices installed
with game. Only those participants are allowed to take part those who are willing
to be a part of this test so as to get the correct data.
3.4 Ethics
Following ethical guidelines are to be followed strictly by the participants of
the survey:
The participants fill in the survey honestly. Participants are required to
provide their age correctly. The participants were briefed about the intention and
method of survey, after the briefing participants have right to decide about
participating in the survey or not. The name of the participants were not
disclosed anywhere [14].
4. Result and Findings In this part we discuss the result and finding of the evaluation technique i.e.
SUS used to test the game: COIN PARTY. The usability goals were kept in
consideration while designing the usability evaluation test [14][15]. There are
three major components used to describe usability: effectiveness, efficiency and the
overall satisfaction [5]. The results of the survey are helpful for drawing attention and
highlighting the issues which needs improvement of game, based on the
recommendation of the respondents for better response in the future.
The usability test objectives are as follows:
Learnability
Various aspects of learnability while playing the game were
studied to know about the load game is putting on the respondent
to memorize the control if he want to use the system in near
future. [2]
Enjoyability Various aspects of user satisfaction while playing the game were
studied to know about the user experience of the game. [4]
International Journal of Pure and Applied Mathematics Special Issue
176
7
GUI Various aspects of GUI were studied to know about the
consistency of the game design. [5]
Task
Performance
Various aspects of user performance were studied during the game
test to know about the Task Performance of the game. [6]
Task
Success
Various aspects of completion of task were studied during the
game test to know about the failure or successful completion of
Task of the game. [6]
Table 2: Usability Test Objectives
5. Results Analysis
5.1 SUS
25 responses were recorded for this survey.
The game was difficult to play for most of the students as in the start of game
there was no help was given, the options were not clearly mentioned, and even the
controls of the game are not clearly visible to them. Respondents start playing game
by hit and trail method, there was a need for instructions on how to play a game [10].
Figure 2: Responses from 25 students for SUS Questionnaire
Figure 3: Average score of students’ response for each question in SUS
12
2018 17
0
24
4
20
5
15
0
5
10
15
20
25
Q 1 Q 2 Q 3 Q 4 Q 5 Q 6 Q 7 Q 8 Q 9 Q 10
1
2
3
4
5
3.122.61
3.17
1.95
2.75
1.55
3.34
1.33
3.01
0.95
0
1
2
3
4
Q 1 Q 2 Q 3 Q 4 Q 5 Q 6 Q 7 Q 8 Q 9 Q 10
Average SUS Score
International Journal of Pure and Applied Mathematics Special Issue
177
8
The game was inconsistent from the other games present. The integration between
levels were not also present, there is an ambiguity for the level on which students
were playing. The switching from lower level to upper level in game was not shown
properly [9]. The total lives present to avail for the game was also not mentioned on
the playing screen. The score in term of coin earned was not shown clearly [10]. This
game put burden on the minds of the user as it will be using memory to memorize the
whole options available for the future use [11]. The power jump controls was not
mentioned properly but when you become use to of playing you will get attach with
the game but for the initiators it was hassle to try different options in order to know
about the correct option for that control. The sounds of the system are too loud and
not pleasing. The color of the overall game is also very loud and not appealing.
Adjective Rating Acceptability Grade Scale
Learnability 50 low F
Enjoyability 65 high D
GUI 45 Not acceptable F
Task Performance 80 Acceptable B
Task success 65 High D
Table 3: SUS scores for Factors
6. Conclusion
Result analysis lead us to the conclusion that there are some points which needs improvement in near future to attract more users. These highlighted points are discussed below.
International Journal of Pure and Applied Mathematics Special Issue
178
9
7. References
7.1. Journal Article J. W. Palmer, "Web site usability, design, and performance metrics," Information Systems Research,
vol. 13, no. 2, pp. 151–167, Jun. 2002.
[1]. A. Bangor, P. T. Kortum, and J. T. Miller, "An empirical evaluation of the system usability scale," International Journal of Human-Computer Interaction, vol. 24, no. 6, pp. 574–594, Jul. 2008.
[2]. A. Sonderegger and J. Sauer, "The influence of design aesthetics in usability testing: Effects on
user performance and perceived usability," Applied Ergonomics, vol. 41, no. 3, pp. 403–410, May
2010.
[3]. S. Wasserkrug et al., "Usability," in Encyclopedia of Database Systems. Springer Science + Business Media, 2009, pp. 3247–3251. [4]. Layla Hasan, Anne Morris, Steve Probets. (2012) A comparison of usability evaluation methods for evaluating e-commerce websites. Behaviour & Information Technology 31:7, pages 707-737. [5]. A. Mouloudi, P. Morizet-Mahoudeaux, A. Valentin. (2011) RAMSES: A Method for the Design Process of Interactive Information Systems. International Journal of Human-Computer Interaction27:2,pages 107-130. [6]. Christos Katsanos, Nikolaos Tselios, Nikolaos Avouris. (2010) Evaluating website navigability:
validation of a tool-based approach through two eye-tracking user studies. New Review of Hypermedia and Multimedia 16:1-2, pages 195-214.
[7]. Kasper Hornbæk. (2010) Dogmas in the assessment of usability evaluation methods. Behaviour &
Information Technology 29:1, pages 97-111.
[8]. Arnold P.O.S. Vermeeren, Jelle Attema, Evren Akar PhD, Huib de Ridder, Andrea J. von
Doorn, Çiğdem Erbuğ, Ali E. Berkman, Martin C. Maguire. (2008) Usability Problem Reports for
Comparative Studies: Consistency and Inspectability. Human–Computer Interaction 23:4, pages 329-
380. [9]. Rajesh, M. & Gnanasekar, J.M. Wireless Pers Commun (2017) 97: 1267. https://doi.org/10.1007/s11277-017-4565-9
[10].Wolmet Barendregt, Mathilde M. Bekker, Don G. Bouwhuis, Esther Baauw. (2007) Predicting effectiveness of children participants in user testing based on personality characteristics. Behaviour & Information Technology 26:2, pages 133-147. [11].Iain Connell, Ann Blandford, Thomas Green. (2004) CASSM and cognitive walkthrough: Usability issues with ticket vending machines. Behaviour & Information Technology 23:5, pages 307-320. [12]. Rolf Molich, Meghan R Ede, Klaus Kaasgaard, Barbara Karyukin. (2004) Comparative usability evaluation. Behaviour & Information Technology 23:1, pages 65-74.
[13]. Deborah Hix and H. Rex Hartsons. Developing User Interfaces: Ensuring Usability through Product and Process. New York: John Wiley and Sons, 1993. 7.2. Book [14]. P. W. Jordan, B. Thomas, I. L. McClelland, and B. Weerdmeester, Usability evaluation in industry. CRC Press, 1996.
[15]. B. Shackel and S. J. Richardson, Human factors for informatics usability. Cambridge University Press, 1991.
7.3. Patent [16]. hide, "Usability metrics," 1998. [Online]. Available:
https://www.nngroup.com/articles/usability-metrics/ . Accessed: Nov. 5, 2016.
International Journal of Pure and Applied Mathematics Special Issue
179
10
Authors
Urooj Waheed
She is currently a Phd scholar at DCS, UoK, having MSCS (2016) in
Computer Science with specialization in Human Computer Interaction
and Intelligent System. She is currently working as Assistant Professor in
College of Computing and Information Sciences of Pakistan Air Force -
Karachi Institute of Economics and Technology and also as visiting faculty in
Department of Computer Science – UBIT. Her main research interests are
Security, Computer Networking, Human Computer Interaction.
Dr. M. Sadiq Ali Khan
He is Chairman and an Associate Professor of Department of Computer
Science – UBIT. He completed his Ph.D in 2011 under the supervision of
Prof. Dr. S.M. Aqil Burney. He had earned his Bachelors and Masters degree
from Sir Syed University of Engineering and Technology. His main areas of
interests are Network Security, Computer Networking, Software Engineering,
Educational Administration, Security Information Technology.
Hira Anwar Khan
She is having MS (2016) in Computer Science with specialization in
Human Computer Interaction and Intelligent System. She is also an MBA
in Supply Chain Management (2014) and MCS in Computer Science. She
is currently working as Lecturer at College of Computing & Information
Science – CoCIS, Pakistan Air Force - Karachi Institute of Economics and
Technology. She is working as academia for about 6 years. Has
researched different domains from CS. Research interest include interaction Design,
Information Systems, Enterprise Integration.
International Journal of Pure and Applied Mathematics Special Issue
180
181
182