usability testing & analysis for commerce€¦ · full sail university - usability -...

USABILITY TESTING & ANALYSIS

Full Sail University - Usability - Hexanalogue 1

Usability Testing & Analysis for Commerce

Mykala Brown, Sam Boyce, Jeremy Boone,

Jonathan Arnold, Colton Auge, Stephen Andruszkiewicz January 30th 2015



Executive Summary This document is an analysis of feedback gathered from participants in a usability test. It’s intention is to aid the developers of Commerce in ensuring that their game meets the high usability standards of consumers. The facilitators of this test played through the game and created impression and exploratory questions to provide beneficial usability information for the developers. The impression questions were designed to find out what the participants thought about the game before they played it. The exploratory questions were designed to get player feedback about the different mechanics and their usability within the game as well as the user interface (UI) elements. A Testing Methodology section shows the developers exactly how each facilitator ran this test and acted as a guideline for facilitators. The survey that each participant took to determine their eligibility for the test is also included in this section. A Training section illustrates how the facilitators trained the participants to participate in the test and the Procedure and Test Steps section detail the specific procedures the facilitators followed when performing the test with the participants. Data collection was performed during the game to measure specific metrics. These metrics were intended to provide information about the difficulty of the game as well how each participant interacted with certain game mechanics. A flow chart was created to aid the facilitators in creating their impression and exploratory questions. It also acted as a useful tool to aid the facilitators and participants in the testing of the game if problems arose. Initial testing was done by facilitators to find bugs or other usability issues with the game. After testing any additional issues that arose were noted. For some major issues recommendations for solving them are made in this report. During testing significant utterances by the participants were noted as well as the location in the game these quotes happened. These quotes were then associated with particular issues in the game. These should help the developers control the emotional responses that players may have toward their game. An Analysis section brings together the answers to the Impression and Exploratory questions as well as the data collected and gives a cohesive overview of the major conclusions reached by the facilitators of this test.



Testing Methodology Each facilitator selected one participant to conduct testing of the usability of the game Commerce. The participants were asked a series of questions prior to testing to ensure they met the criteria for participation (see Qualification Survey below). Each facilitator had a MacBook Pro with a Windows partition. The participants used this computer and partition to run the game for the test. The test took place in the participant’s home. Each test took place in a comfortable, private location. Both during and after testing the participants were asked a series of questions, given verbally by the facilitator.

Participants The participants in this test played the game Commerce in its most current build at the time. They attempted to complete the goal of the game and provided feedback on the game’s usability and user interface. Participants took the following survey to determine their eligibility for this test plan:

1. How many hours per week do you spend playing games? a. 0 b. 1-5 c. 5-10 d. 10-15 e. more than 15

2. What types of games do you enjoy playing? 3. Are you able to proficiently use a laptop computer running Windows?

Participants could be of any age or gender. However, they were required to have the following characteristics:

! Familiar with video games, but not a “hardcore” gamer ○ Plays games more than 0 hours per week but less than 15 ○ Typically plays casual games, for example (Big Fish Games, 2015):

■ Puzzle ■ Casino ■ Hidden Object ■ Board/Card ■ Match 3 ■ Family ■ Arcade

! Able to proficiently use a Windows laptop computer



Training Each facilitator instructed the participants that they were participating in a test of a game that is currently in development. They instructed each participant that they should vocalize their thoughts as they played and informed them that the facilitator may also ask questions during testing and that they should give honest answers/feedback in response to those questions. The facilitators informed the participants that their answers will be recorded. Participants were shown the game and instructed to run it. The participants were then be instructed to complete the tutorial. This provided them with the basic training they needed to interact with the game.

Procedure When the test began, the facilitators asked the participants to find a comfortable place to play the game. The tests were primarily taken in the participant’s home since those were the areas where they normally played games. The participants played until either the game’s objectives were completed, they had played for a period of one hour, or until they no longer felt comfortable playing the game. All three of these potential outcomes were monitored during testing. The test was conducted using the “think aloud” procedure, wherein the participants were encouraged to vocalize their thoughts while playing the game. Facilitators recorded the player’s vocalizations as they played. After they completed the training, the participants proceeded to begin playing the game. Whenever the participants were not vocalizing their thoughts and decisions they were prompted by the facilitator. In these cases the facilitator asked neutral, open-ended questions (e.g. “Why did you go there?”, “Why did you sell that item?”, “Why did you make that decision?”). Once the participants began playing and were in the title menu, the facilitator asked questions about their initial impression of the game. The facilitator also asked exploratory questions as the players reached or completed the relevant portions of the game. While the participants played, the facilitator logged their decisions and collected data on their game experiences. This was done out of view of the participants to prevent biasing their actions. After the participant was finished playing, if needed, the facilitators asked any remaining relevant exploratory questions were not previously asked during the game play. The players were also given a chance to provide any open-ended feedback that they had. The facilitators recorded the answers to these questions as well as all feedback they provided. After the test was complete the facilitator thanked the participants. Test Steps

1. Introduction a. Participants took Qualification Survey b. Participants selected location for test

2. Training



a. Facilitators briefed participants on how the test will be conducted b. Participants opened game c. Participants completed tutorial

3. Test

a. Participant played game b. Facilitators asked impression questions (see Impression Questions) and

recorded answers c. Facilitators asked exploratory questions (see Exploratory Questions) and

recorded answers d. Facilitators recorded participant’s vocalizations e. Facilitators recorded relevant gameplay data (see Data Collection)

4. Post-test Survey

a. Facilitators prompted participants to give any additional feedback b. Facilitators recorded answers c. Facilitators thanked participants



Results Impression Questions 1. What do you think the story or theme of this game is, based on the main menu?

! Old Timey & Medieval. There will obviously be a hero who will most likely be the player. ! Probably fantasy or medieval, with knights and dungeons. ! Trader/Merchant game. There will be vampires. ! Medieval Europe, survival. ! Old times. ! Old medieval role playing

2. Based on the main menu, what sort of genre do you predict this being?

! Probably an RPG. ! An adventure or RPG. ! RPG or shoot ‘em up. ! Looks like a role playing game, reminds him of the role play games on PS3. ! Probably role playing. ! A role-playing game.

3. What do you think the final goal of the game might be?

! To be a master merchant. ! It reminds me of Fable so maybe you’re a hero and you need to save the world. ! Get to the castle (referring to the castle in the BG of the splash screen). ! To find something. ! Something with trading. ! Save the city.

4. What actions do you expect to perform within the game?

! Buying and selling. Option of breaking the law. Combat involved. ! Explore dungeons, fight monsters and get stronger. ! Move around, communicate with NPCs, fight, acquire and sell items. ! Run, pick-up stuff, talk to people, jump, inspect items, inter buildings and houses. ! Trading items. Maybe make deals with other characters. ! Interact with people, do some quests, make money.

5. How complex do you think this game will be?

! Not very complex. The font is pretty corny and the main menu is pretty lame. ! Medium complexity. It might be confusing at first. ! I don’t know, mid-range. ! Looks pretty straight forward. Seems to be very simple. ! It looks like it’s going to be easy. ! Does not look too complex.

6. What demographic do you think the developers are trying to reach with this game?



! RPG players, 13 & up. For all types of players. ! Teenagers or adults who like fantasy games. ! Men aged 12 to 30. ! Young Adults, Older teenagers. Mid – 20s. ! Something educational for kids. ! Teenagers to Young Adults.

Exploratory Questions 1. What do you think about the guard-hiring feature?

! I think its pretty decent. Its something else to add to the game. ! I didn’t get any because I didn’t think I needed them. ! I haven’t seen any danger. ! It would be easier to have if it had a description of the guards and the price ranges before

you go through the process of buying them. You lose more paying for the guards than you do getting items stolen.

! It’s cool, but it didn’t seem to help much. ! They do not seem to have much use.

2. How does the world map compare to world maps in other games?

! I think its pretty trashy. It doesn't match the art. It is more necessary than other games who use it just to fast travel.

! I liked it, it was easy to go back and forth. ! Map is fine, no trouble figuring it out. ! The terrain is realistic, but the towns are hard to spot. The texture of the ocean reminds

me of Feral Hearts, seems like the same texture. No other game really comes to mind, just the map would move faster.

! It’s hard to see where towns are. ! The towns blend into the terrain.

3. Is there something you would change from the shop menu to make it better? If so, what would it be?

! I would like to have the NPCs talk. The amount of money from selling. I would have liked to see animation.

! Make things more obvious, I didn’t know how much I was buying or selling. ! I would prefer it is was in a list form, not pictures. ! Different shops between towns. Having it all look the same is confusing. Images look out

of place. There needs to be a variety of items. It would be nice if the game told you what you originally bought the item for. How much gold so you can tell how much of a profit you are making.

! Use better pictures. ! Make the menus easier to read.

4. How do you feel about the benefit of the tavern rumors?

! Lucky I guess & grateful. ! Liked the idea but I couldn’t see them (Quest completion text bug).



! It’s nothing, I don’t know what to do. It’s a lost cause. ! They haven’t served me well at all, not when the prices change so much depending on

how much of that item you give to that town. ! Didn’t really use them. ! I never used them.

5. Besides getting robbed, what other challenges could the traveler face?

! No profit gain. Not having enough money to travel. ! You can only hold so much, you have to worry about your carry weight. ! I haven’t gotten robbed (he had). Hazardous weather. ! Cart breaking down, lost on the way takes longer to travel, other mundane problems.

Storms when you are on the ocean, shark attacks, or bad winds. ! Not having money to buy items. ! Weight was the major issue I am concerned about.

6. How do you feel about the one day travel time between towns?

! Its too much because you can travel to the next town & its unrealistic. ! I’m not sure, it felt alright. ! I didn’t even notice. ! It’s confusing; it doesn’t say it will change a day. You’ll waste a lot of days. Accidently

going to the map just to check towns will cost you a day which is unfair. ! Pretty indifferent. ! I feel like distance should matter with time rather than money. ALthough I never did travel

between towns. 7. How do you feel about the upgrade system?

! Its useful. i'm just never gonna buy a boat again. Guards are useful. ! I liked it because it meant I could carry more stuff and get protection. ! I would prefer lists over pictures, something less obtrusive. ! Their useless considering the goal of the game. If the goal is just to buy and sell, you

aren’t going to spend money if you can just stay on the main island to meet the goal. ! There’s not much in it. Only used the boat for carrying weight though. ! Everything besides the boat seems useless.

8. How do you feel about the navigation of the map?

! Its pretty good but laggy. ! It was really easy to use. ! It’s OK once you get used to it. ! It is to slow. He likes the scrolling aspect because it makes the world seem bigger.



! Not good. ! I would much rather drag the map.

9. How do you feel about the thirty day time limit?

! I didnt even know there was a time limit. What 30 day limit? JB ! I thought it was enough time but I didn’t really like the time limit. I wanted to play at my

own pace. ! At least it ends. ! Once you get used to the game it would be easy, but the first play through people won’t

understand the system. ! The game didn’t end at 30 days. ! There is a 30 day limit?

10. In your opinion, what is the best strategy when it comes to buying and selling?

! Migrating from town to town near the spawn point to trade and always have guards with you.

! Buy low and sell high. Also write down the prices so you can remember. ! Same as everywhere else. Buy low, sell high. ! The change of prices between towns was drastic which made no sense when they were

right next to each other. ! I ended up going to just the north town and the south town. ! I stood in the same town, bought their stuff and sold it back to them the entire game. SO I

don’t think I ever developed a strategy. 11. What do you think about the game’s overall difficulty?

! Its really, really simple & the age limit should be 6 & up. ! It wasn’t to difficult, I just had trouble because I didn’t know what to do. ! Once you get the hang of it it’s not that hard. It’s a shopping game. ! The goal is average. If you just want to play around, it is very easy. ! It wasn’t really challenging or easy. Just kind of boring. ! It was easy.

12. How does the difficulty using this game compare with other games you’ve played?

! This is one of the easiest games i've ever played. Goes along with tic tac toe and hangman.

! It could of done a better job of explaining things. For example add more hint prompts like ‘You’ve been robbed, by some guards’ or ‘Buy a boat to get across the water.’

! It’s pretty easy to use. ! It is way easier than other games. Doesn’t take much to understand the concept. The

goal is very straight forward, no real challenge. ! Probably the easiest. ! It is definitely the easiest I have ever played.



Testing Issues/Quotes Issue #1 None of the quest / game objectives are

highlighted nor shown to the players.

Type of Problem QA Bug or Game Play Game Play

Severity (Critical, Medium, or Low) Critical

Screenshot

Reproduction steps if QA issue N/A

Solutions if Game Play issue Instruct player on how to play the game and display their objectives.

Quote Are there highlighted quests?

Emotion Frustrated

Game Location After tavern jobs Quote I don’t like how they don’t tell you about missions or where to go!

Emotion Frustrated

Game Location In world map Quote Where do I drop off the coffee? It doesn’t tell me! / I have to buy

the coffee? It didn’t tell me that!

Emotion Frustrated



Game Location Tavern jobs menu Quote They took the coffee? How? Did they pat me down when I entered

the tavern? Do they have X-ray vision? Creeps.

Emotion Annoyed

Game Location Shop Menu Quote Why can’t I finish this job?

Emotion Annoyed

Game Location Shop menu Quote I don’t know where my coffee is.

Emotion Annoyed

Game Location Buying tab Issue #2 Players are never properly notified that they

got robbed, told what was stolen or if soldiers defended successfully.


Severity (Critical, Medium, or Low) Medium



Screenshot


Solutions if Game Play issue Display a message indicating that the player was robbed and list the type and quantity of the items that were stolen.

Quote What are those guys for?

Emotion Confused

Game Location Town travel animation Quote Oh wow I almost got robbed!

Emotion Surprised

Game Location First town travel animation Quote I wonder if robbers take half of your goods or a percentage.

Emotion Confused

Game Location After town travel animation Quote Dang it! I hope my guards took care of that. They better!

Emotion Calm

Game Location After town travel animation



Quote Did I just get mugged?

Emotion Confused

Game Location After robber animation Quote Did my guard come in handy?

Emotion Neutral

Game Location Travelling Quote Can I please NOT get robbed?

Emotion Annoyed

Game Location Robbed animation Quote Am I seriously getting robbed again?

Emotion Annoyed

Game Location Robbed animation Issue #3 There is a bug where the Tavern always

displays the “Mission Complete” dialog after one job is done, new ones then get displayed over this text and are almost impossible to read.

Type of Problem QA Bug or Game Play QA Bug

Severity (Critical, Medium, or Low) Low



Screenshot

Reproduction steps if QA issue 1. Accept Job

2. Complete Job 3. Request Rumor at Tavern

Solutions if Game Play issue N/A Quote I would take whatever ‘I’ item to that town if I could see what ‘I’

item is. / Oh! It was iron!

Emotion Annoyed / Surprised

Game Location Tavern Menu / Shop Menu Issue #4 After tutorial prompts are clicked through there

is nothing that shows the player how to proceed. If the player accidentally clicks without seeing the text they do not know what to do next.





Screenshot


Solutions if Game Play issue Highlight the next step in the tutorial using arrows or by changing the color of the relevant button.

Quote Ok, let’s leave… Why can’t I leave?

Emotion Confused / Annoyed

Game Location Town menu (tutorial) Issue #5 If you sell a certain amount of one item to a

town and it reaches over 50, it will even back out to 12 items of that type and the price will reset.



Screenshot N/A

Reproduction steps if QA issue 1. Sell items to a store until it reaches over 50 stock.



Solutions if Game Play issue N/A Issue #6 When he traveled from Bellmare, the game

showed the traveling cut scene to Shipton but then it booted him back to the main menu. He could move around but could not click on the towns, the roll over text did not work. He was stuck and that ended our play session.



Screenshot

Reproduction steps if QA issue 1. Travel to Shipton

2. Travel to Bellmare 3. Travel to Shipton again

Solutions if Game Play issue N/A Issue #7 Mouse isn’t always responsive in World Map.



Screenshot



Reproduction steps if QA issue 1. Enter World Map 2. Move / Click Mouse

Solutions if Game Play issue N/A Quote This is so slow. / Check back in five years, maybe I would have

arrived by then.

Emotion Annoyed

Game Location World map Quote This map is weird to move

Emotion Confused

Game Location World map Issue #8 Can travel to any town, regardless of money

available.



Screenshot

Reproduction steps if QA issue 1. Spend cash to drain below required

cost to travel



2. Travel to town that costs more money than available

Solutions if Game Play issue N/A Quote What’s the point? If I had known that I’d just double click the

whole time!

Emotion Annoyed / Frustrated

Game Location World map Issue #9 Goal of the game is never properly explained.



Screenshot


Solutions if Game Play issue Add another step in the tutorial explaining that the player has to reach 2,000 gold within the thirty day time limit or add a section in the world map stating how many days you have left and how much gold you need for example the empty space where the cursor is in the



screenshot. Quote I didn’t realize there was a goal.

Emotion Surprised

Game Location World map



Issue #10 Upgrades cost way too much gold



Screenshot


Solutions if Game Play issue Reduce the costs of upgrades or increase item selling price to compensate.

Issue #11 Player can win the game without having to

leave Fellsmore at the beginning.



Screenshot N/A

Reproduction steps if QA issue 1. Enter Fellsmore 2. Buy towns stock 3. Sell towns stock 4. Leave 5. Repeat

Solutions if Game Play issue N/A Issue #12 Error messages remain after switching to a

new tab or exiting to the world map.





Screenshot

Reproduction steps if QA issue 1. Buy items until there are either none

left, you are out of money, or you have no weight left.

2. Switch to another tab in the town or press the leave button.

Solutions if Game Play issue N/A



Known Developer Issues

! The game will sometimes lag out in the beginning preventing the player from being able to pan the screen.

! Once in a while the towns will disappear in the tutorial at the start. ! Occasionally you won’t be able to mouse over a town or click on one in the tutorial. ! Mouse scrolling and the ability to click on towns or escape the game may cease to

function after the snow robbery matinee is played. ! After accepting a job the message “Thanks friend and best of luck” may be permanently

displayed in the dialog box and will obscure any other dialog. ! There is no feedback informing the player that they have successfully completed or failed

a job. ! If a player accepts a job there is nothing to remind the player later what jobs they have

taken. ! While hiring guards if the player clicks on the back button instead of the purchase button

while the total cost is displayed, and then clicks on the button to hire guards the player will be able to hire guards 1 at a time for 0 gold.

! In Norwich the player could amass a fortune by buying silver from the town, reselling it there immediately, leaving town and reentering it to buy more.

! In Wolford the player could amass a fortune by buying steel from the town, reselling it there immediately, leaving town and reentering it to buy more.

! The player will permanently retain any guards they hire.



Data Collection Demographics

Test Subject Gender Age Gaming Experience

Weekly Play

1 Male 14 FPS, Role Playing,

RTS 1-5 Hrs

2 Female 54 Adventure, RPG, Animal Crossing 5-12 Hrs

3 Male 50 Card, Puzzle, Arcade 5-12 Hrs

4 Female 19 Some FPS,

Adventure, Puzzle 1-2 Hrs

5 Male 23 Role Playing and

Racing 1-2 Hrs

6 Female 19 Role Playing and

Puzzle 5-12 Hrs

Analysis Player turns to $2000



In the six play tests conducted only three of the six play testers managed to reach 2,000 gold. On top of that only one of the play testers reached 2,000 within the thirty day time limit. The three play testers who were unable to reach 2,000 either had their progress stopped by a bug or they simply ran out of gold. With this evidence it is safe to say that almost all of the play testers struggled reaching the goal and this problem should be looked into further. Some solutions may be lowering the required goal, giving the player more time to complete the goal, or adjusting prices to aid the player in making gold. Game play time to $2000 Only three of our six playtesters were able to reach 2,000 gold and they did so in an average time of 26 minutes. The exact times, however, are pretty diverse, reinforcing the amount of player choice within the game. Our fastest play tester managed to reach it as quickly as 15 minutes whereas our slowest playtester didn’t reach it until 38 minutes. While there is a set goal this data shows that players are able to reach it at their own pace and can even gain some replay value by trying to beat their previous times. Total Funds on Day 30 Across the board, none of the players were able to reach the goal of $2,000 by day thirty. The one player who was successful only reached the goal by exploiting a bug in the game. Most were unaware of what they had to do or that there was even an end goal to the game. One player ran out of funds and he was still able to travel around the game through a glitch. The game did not inform any of the players by day thirty if they had lost or won. We noticed that the players were trying to earn enough gold to get winter gear and the boat so they weren’t completely focused on saving as much as they could. All team members allowed their play testers to keep playing the game until the hour was over or once they felt the play session had gone on long enough. Once play testers were informed about the thirty day goal, many said they would have just traded between three or four cities. Two stated that they wouldn’t have bothered with the upgrades that it cost more just traveling there than actually getting any real profit. Not a single player knew that they had an overall goal they were supposed to be achieving; many said they were unaware of this. Number of instances of travel In the six play tests conducted almost all of the play testers decided to travel to a new town almost every turn. Only one play tester didn’t since they found out that they were able to repeatedly buy and resell items from a single town. Other than this issue all of the testers traveled to a different town every turn. In general, they chose to visit the same set of towns over and over instead of exploring the different sections of the map as there was no major benefit to attract the player to new towns. Because player’s were not all fully aware of the penalty associated with being robbed they were not always weighing the risk/reward of travelling. Number of items bought



In the six play tests conducted the number of items bought has been consistently much higher than the items sold at a total of 1,337 items bought. On average, players purchased 33.9% more items than what they were able to sell. This is due to the high rate at which players are robbed between towns (see Instances of Travel). Number of items Sold From six play tests, there were a total of 1068 items sold which is a solid 269 items (or a whole 25.2%) lost to robberies. This even includes players who had purchased guards to protect them from said robbers. After players made potential purchases for upgrades they were left with a total of $6,348 between the five play tests, leaving each player with an average of $1058 per game which is 47.10% short of reaching the goal of $2000 on average. In these six play tests, only one player succeeded in reaching that goal. As players are losing such a large percent of their items to sell, it would appear that either guards should be made more effective, the rate of robberies should be decreased, or more information should be provided to the player so that they can weigh the risks of their actions.

Game Play time to 30 Days Within the six play tests the average time it took to reach the 30 day mark was coincidently 30 minutes. Most of our play testers hit 30 days around the 20 minute mark. However, two of our play testers didn’t finish until 41 and even 57 minutes. The diversity of these times is a good thing and it shows that the game allows players to either rush through the turns or take their time and plan out a strategy to make the most gold. On top of that if players do take their time they will still finish in an appropriate amount of time. Day 30 seems to be a good stopping point for casual players, they can pick up the game and play it without having to worry about investing a large amount of time in order to complete it. Average Gold earned per minute / per day The disparity between different players’ earned gold, when broken down by minute and by turn, tells an interesting story about the different game play types these players adopted. For example, there were three participants that earned around 55-60 gold per minute. However their range for gold earned per turn is large. Those participants that earned less gold per turn adopted a strategy where they would single out one item for purchasing and look for the best deal they could get for that item. They would travel often but buy and sell less. The players that earned more per turn would generally max out on items and then sell them the first place they could make a profit. They did not travel as much but bought and sold more items. The outliers had a range of issues. The player that earned very little in terms of both time and turns simply took their time when making decisions and was a little slower to process the game than the average player. The player that zeroed out their funds went all for broke purchasing upgrades. One player exploited a bug to achieve their results. Player win versus loss



After analyzing all of the test results gathered from the participants we can conclude with confidence that the vast majority of future players will most likely fall short of successfully achieving the game objective in the current build. We believe that this could be better resolved by providing players a clear way to check objectives and better track their progression. Also, if the cost of upgrades, the percentage of profit gained from trading, and the frequency of theft was better balanced it will greatly improve the chances of better player choices and success.



Recommendations Least enjoyed game features The overall concept of the game was rather interesting and simple to understand. However there were a number of critical elements of the game that were flawed and in most cases caused mass confusion amongst our play testers. The confusion started right as they started playing the game since there was absolutely no directive as to what to do next nor was there a clear indication of where they were located in the world map. Our recommendation would be to not only highlight what player's overall game goal is but add some interesting lore to help immerse players into a more meaningful story. The tavern features were also a major sticking point for players. These features had a number of issue with their interactivity that caused frustration on the part of the testers. One is the known bug that causes text overlay. Another is the fact that the removal of the quest item and inserting of the reward is done without alerting the player as to what is happening. This lack of visibility makes it difficult for the player to interact with the job feature. The rumors feature also seemed to be repetitive and was almost always ignored by players. Perhaps increasing the rewards available from these rumors as well as adding some variety will guide players to this feature. Most enjoyed game features Players enjoyed the buying and selling features the best. They had the most fun searching for bargains and trying to buy low/sell high. To aid in this feature we would recommend implementing a inventory system where the player could view all the items they are carrying with them. If they could also see how much they paid for each item that would be all the better. Perhaps there could be a log or ledger that showed the player all their past transactions. This would help them figure out where to buy and sell and give them more “recognition” over “recall” information (Nielsen, 1995). As the game stands now a player must spend a day to see what items they have available for sale; if his/her information was available in one location the player could make better decisions. Players also enjoyed the upgrade system. However, these upgrades were not always visible to them. Perhaps instead of using a counter to show the players they have 1 or 0 boats/winter gear those items could become highlighted in the World Map screen to show the player they have access to them and thus access to more areas of the map. Having towns that are available for travel colored differently than towns the player cannot travel to on the World Map would be another way to relate this information to the player.



Post Mortem What went correctly? Overall, none of us really had problems when it came to finding a play tester as we all had family or friends that were willing to test the game out. Much to our convenience many of them were not avid gamers. All of them were willing to put up with the ‘Think out loud’ method even if many paused trying to unfocused from the game to give us an answer. Putting together the play testing method wasn’t hard on us at all. We knew what we had to do and all of us were on the same page when it came with putting a structured plan together and to perform this plan. Splitting the responsibilities amongst the team members was extremely easy as everyone stepped up to the part or simply did things without being asked. If there were technical issues, team members were willing to do another job to make up the difference. No one person was carrying the weight of the team which allowed for a peaceful work environment. What went incorrectly? When brainstorming ideas of how this play testing should go or coming up with questions to ask, some of us were thinking too much about and were making it more complex than it really needed to be. Often the rest of the team had to reel them back in, making them realize that it was much simpler than they were thinking. The biggest challenge we had was with coming up with questions. With the sheer size of our team, we kept running into the problem that two team members were coming up with the same idea or at least close to it. It took awhile to figure out what the other was trying to ask the play tester. It also took awhile to reword the questions, making sure that we were not repeating ourselves. We didn’t want to have the player becoming too annoyed with the amount of questions we would be asking them for such a short game. Our group size also meant that we encountered a wide range of issues with the game, both similar and completely different from one another. It took awhile to go through each and every issue, trying to figure out if it was a QA bug or game play. We felt that we kept stepping on each other’s feet with ideas and suggestions because there were so many voices that needed to be heard. Granted this brought a lot of new ideas to the table, but it proved difficult trying to hear everyone out. What did we learn from this activity? Sometimes stepping away from a project simply isn’t enough, you need another set of eyes that hasn’t been looking at the project day in and day out to notice details you have missed. We discovered that throughout the play testing this week, regardless of the fact that we have six people on our team, that the play testers found problems we hadn’t even noticed the week before. It just goes to show that no matter how many times you look over and play test your own game, that you can still miss important errors.



The problem with having a group so large is that we had eighteen questions to ask throughout the play testing. We did try to plan ahead, trying to set up the questions in the order we thought the players would encounter certain events, but we can only guess. Each of us had to figure out when it was appropriate to ask the list of questions. Some of us even came up with other questions throughout the game testing because the play testers did run into issues that we hadn’t encountered before. This opened our eyes to the fact that we have to be flexible and come up with questions on the spot without seeming surprised ourselves. The idea of play testing is simple enough but once we started to think about all the actions involved it proved harder than we thought. It isn’t just sitting a player down and asking them to play a game. You need to think about what you want to ask them, how long the player should play the game to get a good feel for what it is about, what sort of environment you want the testing to occur, and as we already discussed when to ask the questions. We as a team ran through the game several times ourselves making sure we knew the game inside and out before performing the play tests. We all determined that a team needs to know their game well enough to be able to think of questions on the spot, to debug any sort of errors the play tester might run into, and to assist if needed. This past week really opened our eyes to how much work is involved with play testing.



References Big Fish Games. (2015, January 1). Retrieved January 22, 2015, from http://www.bigfishgames.com/download-games/top-pc-games.html Nielsen, J. (1995, January 1). Nielsen Norman Group. Retrieved January 12, 2015, from http://www.nngroup.com/articles/ten-usability-heuristics/

usability testing & analysis for commerce€¦ · full sail university - usability -...

Documents