test plan for commerce

USABILITY TEST PLAN

Usability Test Plan

Version 1.0

Mykala Brown, Sam Boyce, Jeremy Boone,Jonathan Arnold, Colton Auge, Stephen Andruszkiewicz

January 24th 2015

Full Sail University - Usability - Hexanalogue

USABILITY TEST PLAN

Executive Summary

This document is a test plan created to gather feedback from participants to aid the developers of Commerce in ensuring that their game meets the high usability standards of the consumers.

The facilitators of this test played through the game and created impression and exploratory questions to provide beneficial usability information for the developers. The impression questions are designed to find out what the participants think about the game before they play it. The exploratory questions are designed to get player feedback about the different mechanics and their usability within the game as well as the user interface (UI) elements.

A training section has been added as instructions for the facilitator on how to fully disclose and guide the participant along the test. The procedure and test steps section following afterwards details the specific procedure in which the facilitator will test the participant.

A testing methodology section was created as an outline so that the developers know exactly how each facilitator is running this test and acts as a guideline for said facilitators. This section also lists the equipment and environment that each facilitator will be utilizing. The survey that each participant will be taking to determine their eligibility for the test is also included in this section.

A data collection sheet was created as an easy way for the facilitator to store gameplay information while the participant is testing the game. This is provided in a separate excel document.

A flow chart was created to aid the facilitators in creating their impression and exploratory questions. It also stands as a useful tool to aid the facilitators and participants in the testing of the game if problems may arrive.

The initial testing was done as a quick bug analysis that the facilitators found in their initial playthrough of the game. These bugs may help the developers for their next iteration of their game.


USABILITY TEST PLANTesting Methodology

Each facilitator will select one participant to conduct testing of the usability of the game Commerce. The participant will be asked a series of questions prior to testing to ensure they meet the criteria for participation (see Participation Survey below). Each facilitator has a MacBook Pro with a Windows partition. The participant will use this computer and partition to run the game for the test. The test may take place in the facilitator’s, participant’s, or a third party’s home but will take place in a comfortable, private location. Both during and after testing the participant will be asked a series of questions, given verbally by the facilitator.

Participants

The participants in this test will play the game Commerce in its current build. They will attempt to complete the goal of the game and provide feedback on the game’s usability and user interface.

Participants will take the following the survey to determine their eligibility for this test plan:

1. How many hours per week do you spend playing games?a. 0b. 1-5c. 5-10d. 10-15e. more than 15

2. What types of games do you enjoy playing?3. Are you able to proficiently use a laptop computer running Windows?

Participants can be of any age or gender. However, they must have the following characteristics:● Familiar with video games, but not a “hardcore” gamer

○ Plays games more than 0 hours per week but less than 15○ Typically plays casual games, for example (Big Fish Games, 2015):

■ Puzzle■ Casino■ Hidden Object■ Board/Card■ Match 3■ Family■ Arcade

● Able to proficiently use a Windows laptop computer


USABILITY TEST PLAN

Training

The facilitator will instruct the participant that they are participating in a test of a game in development. They will instruct the participant that they should vocalize their thoughts as they play and inform them that the facilitator may ask questions during testing and that they should give honest answers/feedback in response to these questions. The facilitator will also inform the participant that they will be recorded while playing.

Participants will be shown the game and instructed to run it. The participants will then be instructed to complete the tutorial. This will give them the training they need to interact with the game.

Procedure

When the test begins, the facilitator will ask the participant to find a comfortable place to play the game. If the test is taking place in the participant’s home they will be instructed to play the game in an area they normally play games. The participant should play until the game’s objective is complete; for a period of one hour, or until they no longer feel comfortable playing the game, whichever happens first.

The test will be conducted using the “think aloud” procedure, wherein the participant is encouraged to vocalize their thoughts while playing the game. Facilitators will record the player’s vocalizations as they play. After completing the training, the participant will begin playing the game. If the participant is not vocalizing their thoughts and decisions they may need to be prompted by the facilitator. In this case the facilitator should ask neutral, open-ended questions (e.g. “Why did you go there?”, “Why did you sell that item?”, “Why did you make that decision?”). When the participant begins playing the facilitator should ask questions about their initial impression of the game. The facilitator should also ask exploratory questions as the player reaches or completes the relevant portions of the game.

While the participant plays the facilitator will log their decisions and collect data on their game experience. This should be done out of view of the participant to prevent biasing their actions.

After the participant is finished playing, the facilitator will ask any relevant exploratory questions that were not asked during the game play. The player should also be given a chance to give any open-ended feedback they may have. The facilitator will record their answers to these questions and any feedback they give. After the test is complete the facilitator should thank the participant.


USABILITY TEST PLAN

Test Steps

1. Introductiona. Participant takes Participation Surveyb. Participant selects location for test

2. Traininga. Facilitator briefs participant on how the test will be conducted b. Participant opens gamec. Participant completes tutorial

3. Testa. Participant plays game (until complete, for 1 hour, or until they are no longer

comfortable playing)b. Facilitator asks impression questions (see Impression Questions) and records answersc. Facilitator asks exploratory questions (see Exploratory Questions) and records answersd. Facilitator records participant’s vocalizationse. Facilitator records relevant gameplay data (see Data Collection)

4. Post-test Surveya. Facilitator prompts participant to give any additional feedbackb. Facilitator records answers

Facilitator thanks participant


USABILITY TEST PLANImpression Questions

1. What do you think the story or theme of this game is, based on the main menu?

2. Based on the main menu, what sort of genre do you predict this being?

3. What do you think the final goal of the game might be?

4. What actions do you expect to perform within the game?

5. How complex do you think this game will be?

6. What demographic do you think the developers are trying to reach with this game?

Exploratory Questions

1. What do you think about the guard-hiring feature?

2. How does the world map compare to world maps in other games?

3. Is there something you would change from the shop menu to make it better? If so, what would it be?

4. How do you feel about the benefit of the tavern rumors?

5. Besides getting robbed, what other challenges could the traveler face?

6. How do you feel about the one day travel time between towns?

7. How do you feel about the upgrade system?

8. How do you feel about the navigation of the map?

9. How do you feel about the thirty day time limit?

10. In your opinion, what is the best strategy when it comes to buying and selling?

11. What do you think about the game’s overall difficulty?

12. How does the difficulty using this game compare with other games you’ve played?


USABILITY TEST PLANData Collection

Facilitators will gather the following data about their participant’s game usage during their test:● Player turns to $2000 ● Gameplay time to $2000● Total funds on day 30● No. of instances of travel● No. of Items bought● No. of Items sold● Gameplay time to 30 Days● Average Gold earned per minute● Average Gold earned per turn (day)● Player win/loss


USABILITY TEST PLANInitial Testing During the initial playthrough of the game the facilitators ran across a wide array of problems such as design issues, bugs, and game crashes. The facilitators marked down these problems as they came across them in hopes of aiding the developers in future iterations of the game. The complete list of problems can be found below.

● Completing a job at the tavern permanently freezes the completion text In the barkeepers dialogue box rendering it impossible to read any rumor text.

● Boat and Winter Gear only cost 1 gold.● When travelling to a new town there is a chance that the trading menu will not open locking the

player in the map screen permanently.● Game does not end. Still going at 101 days.● No UI to inform the player whether a robbery was successful or defended by guards.● No information about what was robbed after robbery.● Game becomes unresponsive after completing the tutorial.● Goal of the game is never properly explained.● Hitting Escape sends you back to the menu with no warning, causing you to lose all your

progress.● Able to travel to a town even if you can’t afford the travel fee.● Map is sometimes unresponsive to mouse movement input.● If a player happens to reach zero gold, the game has no default setting to tell the player that they

lost. The game will keep playing even if you are at zero gold.● The mouse lags making it difficult to navigate the game.● The barkeep does not inform the player the direction of the town they need to deliver good too.

Player must scroll around the entire map pretty much playing I spy trying to find the town.● Traveling to any town costs the same amount of days no matter if it is across the town or across

land.● Game sometimes crashes after a robbery.● Game ran at a constant 10fps on MacBook Pro Windows partition.


USABILITY TEST PLAN


USABILITY TEST PLANReferences

Big Fish Games. (2015, January 1). Retrieved January 22, 2015, from http://www.bigfishgames.com/download-games/top-pc-games.html


test plan for commerce

Documents

game commerce

test steps section

testing methodology

initial testing

games usability

exploratory questions

impression questions

series of questions