game production: testinggameproductionuu.files.wordpress.com/2014/01/2015-18-testing.pdf3 1. intro...

1

Game Production:

testing

Fabiano Dalpiaz

[email protected]

mailto:[email protected]

2

Outline

1. Intro to game testing

2. Fundamentals of testing

3. Testing techniques

Acknowledgement: these slides summarize elements from chapters 1 to 3 and 6 to 13 of the book “Game testing: all in one” by Charles P. Schultz, Robert Bryant and Tim Langdell –2005, Course Technology, ISBN:1592003737

Lecture contents

3


Game software is complex

People write game software and people make mistakes

Software tools for building games are not perfect

There is a lot of money at stake for games to succeed

Multiple platforms, configurations, devices

People expect more out of every game you make

Paying customers don’t like bugs

Critics are standing by ready to rate your game

Why is testing important?

4


Two rules of game testing

Rule 1

Rule 2

5


You encounter game project panic when you are

Unfamiliar

Unprepared

Under pressure

Unrested

Nearsighted

… quite often, this corresponds to crunch time

Luckily, there are systematic techniques to get prepared to crunch time

Rule 1: don’t panic

6


You need to test something when that something cannot be trusted

If you trust component X, you don’t need to test it

Do not trust developers

“Only a few lines of code have changed, so don't inspect any other lines”

“The new audio subsystem works the same as the old one, so you only need to run your old tests”

It is your responsibility to provide evidence that it works

Find the balance between test and trust: why?

Rule 2: trust no one

7


A sequence of activities to characterize game testing

Play the game

Identify bugs

Amplify problems (generalize it)

Notify the team

Testify to others

Verify the fix

The PIANo TV sequence

8


Playing to crash it

As opposed to playing to have fun

Testing can be fun… but testers don’t choose what to test, when, and how to play

All play is done with a purpose

Explore an area of the game

Check the enforcement of certain rules

Look for a specific kind of problem

Follow the instructions

Repeat the same sequence

a. Playing the game

9


Testing has two purposes

Find defects in the game code and design

Demonstrate which parts of the game are working well

Given a test

The test finds no problem PASS

The test finds a problem FAIL

Cannot even get to relevant part of the test BLOCKED

Important: passing all tests does not mean that the game has no bugs

Why?

b. Identifying bugs

10


Not all testers will find the same bugs

The effect personality type (Myers-Briggs Type Indicator)

Judger: you prefer a structured, ordered, and fairly predictable environment

Perceiver: you prefer to experience as much of the world as possible

b. Identifying bugs: tester personality

11


Objective: narrow down the scope for the developers

Help them localize the problem

Get the defect fixed right the first time

Reduce overall time and cost spent on the problem

Key principles to do so

Early bird detection: identify bugs as soon as new features are introduced (… don’t wait crunch time)

Find “all” places in the game where the bug occurs

• Not only in the place you identified it first

Increase the frequency

• Eliminate unnecessary steps to get it appear

• Find more frequent scenarios

c. Amplifying problems

12


Record the information and notify the developers

Typically, via software tools

Bug tracking systems

It is essential to deliver an accurate report

Proper title

Informative description

Type of bug: crash, rendering issue, change request, …

Priority

Replicability (How? Always?)

Any other useful information (e.g., screenshots, logs, videos)

d. Notifying the team

13


Function error: one that affects a game capability or how the user experiences the game. The code providing this function is missing or incorrect in some or all instances

E.g., missing code to activate turbo in a car racing game

Assignment: it is the result of incorrectly setting or initializing a value used by the program or when a required value assignment is missing

E.g., scoreboard in a soccer game initialized at 1-1

Checking: the code fails to properly validate data before it is used

E.g., using “=” instead of “==”

d. Notifying the team – defect typing

14


Timing: has to do with the management of shared and real-time resources

E.g., the bg music of level 1 does not stop when level 2 starts

Build/Package/Merge: the result of mistakes in using the game source code library system, managing changes to game files, or identifying and controlling which versions get built

E.g., code from two incompatible versions are compiled

Algorithms: faults in the developed algorithms

E.g., NPC freezes, either due to unforeseen situation or due to excessive “thinking”


15


Documentation: occur in the fixed data assets that go into the game

E.g., grammar errors and typos, empty help screen

Interface: occurs at any point where information is being transferred or exchanged

E.g., passing the wrong value of one argument


16


Testers get emotionally attached to their found defects

… but the final decisions are made by others

The project or development manager, typically

The change control board, in large organizations

… these people are not as much attached to the defects

How are these decisions made?

Prioritization: are there more urgent bugs?

Crash or not?

Fixing too expensive

A certain amount of bugs may be acceptable

… a business decision, in the end

e. Testify to others

17


Eventually a fix will be created (not always though)

What do testers do then? Repeat the same sequence of steps to check that the bug has been correctly fixed

… note that this may introduce new bugs!

f. Verify the fix

18

2. Fundamentals of game testing

Basic quality means the game should be fault-free

Not the only factor though

What else determines the quality of a game?

Quality of the story

Quality of the game mechanics

Quality of in-game audio and visual effects

Beauty of the visual style

Use of humor and exaggeration

NPC Artificial Intelligence (AI)

Clear interface

Ease of use

Compatibility with target platform (console, smartphone, …)

Ensuring quality

19


The actual quality of a game is defined by its design and code implementation

How to appraise the quality?

Testing is a prominent technique…

… but not the only one

Testing is expensive!

“Soft” techniques to catch defects as they are introduced

Walkthroughs (up to 10 people)

Reviews (5-6 people)

Checklist-based reviews (author-reviewer)

Inspections

Adherence with standards (user interface, code naming)

Appraising quality: not only testing

20


Sigma level: defectiveness of code relative to its size

Defects per million lines of code

Lines of code are normalized, according with the language

• 1 line of C code = 3 lines of assembly code

• 1 line of Perl code = 15 lines of assembly code

Sigma 6 = outstanding

Sigma 5.5 = very good

All depends on how many defects are actually found!

Measuring quality: sigma level

21


Phase containment: success of the team at finding defects in the code

Measure: phase containment effectiveness (PCE)

Faults identified in the phase they were introduced are known as errors

Typically found by walkthrough, reviews, testing

Faults identified in later phases are known as defects

Typically found by customers

PCE = #in-phase faults / (#faults in that phase and subsequent phases)

Measuring quality: phase containment

22


Most game testing is black-box: testing is done from outside the application

No knowledge of the source code is required

Black-box testing

23


White-box testing gives the tester the opportunity to look into the source code

Extremely difficult, due to the complexity of the code

This is done for certain situations

Tests performed by developers prior to integrating new code

Modules that will become part of a reusable library

Methods/functions that are essential part of a game engine

Parts that can be used by modders

Low-level routines to interact, e.g., with new hardware

White-box testing

24


Test case: a single test performed to answer one question

Test suite: a collection of test cases

Every tester is assigned a set of test suites

He/she is responsible for them throughout the project

In its simplest form, a test suite is a series of test cases that the tester can perform sequentially

Typically, test suites are more elaborated

IF/THEN/ELSE

LOOPS

Test cases and test suites

25


The overall testing phase shall be managed continuously

Just like other phases of (game) production

What to check?

Are testing suites effectives?

Are testers effective?

Are we successfully tackling all severe defects?

Is testing on track?

What to do when achieving bad results?

Apply compensation plans

Typically defined before that through risk assessment

Managing testing

26


Testing is a trade-off between cost and amount of testing

Combinatorial testing is a good mix

Pairwise combination: each value you use for testing needs is combined at least once with each other value of the remaining parameters

Take the following example with three parameters

Gender = Male, Female

Light saber = One handed (1H), two handed (2H)

Force = Light, Dark

Combinatorial testing

27


A full combinatorial table would require 8 tests

… exponential in the number of parameters and possible values per parameter


28


A pairwise combinatorial value shall define one test case for every pair of input parameters

Male Gender paired with each Light Saber choice (1H, 2H)

Female Gender paired with each Light Saber choice (1H, 2H)

Male Gender paired with each Force choice (Light, Dark)

Female Gender paired with each Force choice (Light, Dark)

1H Light Saber paired with each Force choice (Light, Dark)

2H Light Saber paired with each Force choice (Light, Dark)


29


Graphical models representing game behaviors from the player’s perspective

Testing = travel through the diagram

Every TFD consists of elements

Flows connecting states

Triggering events

Actions to be performed

Special states: IN (start) and OUT (end)

Semantics: when a certain event occurs, and currently in state S, execute a certain action to reach state S’

Test flow diagrams (TFD)

30


Basics: start from the IN state, create a set of states and flows that lead to the OUT state

Example: consider the feature of picking up a weapon and its ammo while the game properly keeps track of ammo count and performs correct audible and visual effects

Designing a TFD

31


When the player gets a gun, gun effects must be played, and the player has a gun

Designing a TFD

32


When the player gets ammo, ammo effects are played, and the player has ammo

Designing a TFD

33


… supporting dropping ammo and gun

Designing a TFD

34


… and getting ammo/gun when the player has gun/ammo already

Designing a TFD

35


Support shooting all ammos, get loaded gun, …

Designing a TFD

36


Define the acceptance criteria for each event/action

Action “AmmoEffects”

Check that the Bio-Rifle ammo sound is made

Check that the game temporarily displays "You picked up some Bio-Rifle ammo" in white text above the gun icons at the bottom of the screen

Check that the temporary text on the display fades out slowly

Event “DropGun”

Hit the "\" key to drop your selected weapon

Designing a TFD

37


Test path: a series of flows, specified by the flow numbers in the sequence in which they are to be traversed

… an individual test case

Paths begin at the IN state and end at the OUT state.

A set of paths provides behavior scenarios appropriate for prototyping, simulation, or testing

Many paths are possible for one TFD!

So, which paths should testers execute?

Minimum path generation: every flow is followed 1+ times

Expert path generation: based on the expertise of testers

TFD paths

38


Use minimum path generation: every flow is followed 1+ times

TFD paths: exercise

39


Depends on the type of functionality to test

The following table provides some guidance

Combinatorial testing or TFDs?

40


Cleanroom testing: exercise software to determine mean time to failure (MTTF) measurements over the course of a development project

Applied to games to assess why customers find problems even after games were tested for thousands of hours

Essence of cleanroom testing: produce the tests that play the game the way players will play it

Objective: prevent the most common problems that players would encounter

Cleanroom testing

41


Different usages imply using different game elements

Mode-based usage

Single player, campaign, multiplayer, online

Player-type usage

Achiever, explorer, socializer, killer…

Real-life usage

Log usage from real players

E.g., from previous editions, similar games, or early builds

Cleanroom testing: usages

42


Cleanroom tests can be created for any testing method

Required input: usage probability

To be assigned to each step in the test

Example: Halo advanced controls

Features Usage probabilities per feature

Cleanroom testing: generation

43


Generated tests (example, no details)


44


An example for cleanroom TFDs

The figure shows probabilities

One can generate test paths only for most likely flows (or start from those)

In inverted usage, emphasis on less frequent functions


45

Mandatory literature

Memon, Atif M., Martha E. Pollack, and Mary Lou Soffa. "Hierarchical GUI test case generation using automated planning." Software Engineering, IEEE Transactions on 27.2 (2001): 144-155.

Nie, Changhai, and Hareton Leung. "A survey of combinatorial testing." ACM Computing Surveys (CSUR) 43.2 (2011): 11.

game production: testinggameproductionuu.files.wordpress.com/2014/01/2015-18-testing.pdf3 1. intro...

Documents