Game Design Processes and Principles

Prototypes and Evaluation

“An incomplete design” (Preece et al, 1994)

“Artefacts that simulate or animate some but not all features of the intended system” (Dix et al, 1998)

Prototypes

What are Prototypes

1. Throw away

Design knowledge gained is used to build final product, but prototypeis discarded.

Preliminaryrequirements

BuildPrototype

EvaluatePrototype

Final requirementsAdequate?yesno

Three Approaches to Prototyping

what are prototypes?

2. Incremental

Three Approaches to Prototyping

System is developed incrementally as separate components. Prototypes are developed and evaluated for each component in turn.

Identifycomponents

Designing component / Prototype

Operation andmaintenance

Systemcomplete?

no yes

deliver increment deliversystem

what are prototypes?

3. Evolutionary

Three Approaches to Prototyping

Prototype is not discarded and serves as the next iteration of design

Build prototype EvaluatePrototype

Amend prototype

Final requirements

Adequate?yesno

Preliminaryrequirements

what are prototypes?

What to prototype?

what are prototypes?

Floyd (1984) distinguishes between the breadth of evaluation

Horizontal Prototyping (broad/shallow)Demonstrates a broad spectrum of a product’s features, but withoutextensive functionality.

Vertical Prototyping (narrow/deep)

Demonstrates the exact functionality of a product, but only for a smallsection of the entire release.

- i.e., developing a single level of a game from which you extrapolate

developing the spell-checking function of a word processor

- Useful for GUI design

Why prototype?

why prototype?

CostIssues caught at prototype stage cheaper and easier to implement thanif reported late or post-production.

Product conceptulisationUseful to gain a better understanding of the game design – for you,team mates, the publisher and the target audience.

Risk reductionEasier to gain further finances if you can show prototype is enjoyed.

Proof of conceptDemonstrates that a particular technology/game mechanic/team works.

EvaluationFacilitates early evaluation of product.

Why NOT prototype?

why prototype?

User recruitment Recruiting and managing users for prototype evaluation is time consuming, and involves much effort

TimeCan be viewed as taking precious time away from ‘real design task’.

CostAdds 5-10% on pre-production costs.

Project managementDifficult to manage iterative design projects. Hard to predict the number of Iterations needed, the costs involved, and specification creep more likely.

Approaches to prototyping

how to prototype

Software PrototypesEntirely digital, relying on computer code to function.

Popular products and languages for software prototyping include:

Game Salad or Game MakerBlitz Basic

UnityFlash

Visio (for flowcharts/maps)Excel (for spreadsheets)Access (for structuring data)

Products

Languages

Level Editors (UnrealEd, WarCraft III, etc)

Approaches to prototyping

how to prototype

Physical Prototypes- Typically paper-based prototyping used in game design

Useful tools for physical prototyping include:

- Applied early in the design phase, usually prior to software prototype

Paper and card (of different colours)Hexagonal paper (http://www.incompetech.com/beta/plainGraphPaper/)PensPritt StickLarge RulerScissorsFigures (lead, plastic, or stolen from other games)

- Helps you to build game structure, determine how game elements interact, and formulate a systematic approach to game functionality.

Approaches to prototyping

how to prototype

Physical Prototypes

Approaches to prototyping

how to prototype

Physical Prototypes

1st Person Shooter

1. Take a sheet of hex paper

2. Mark spawn points in red

3. Add lines to represent walls

4. Add your units

Approaches to prototyping

how to prototype

Physical Prototypes

1st Person Shooter

Each player gets the following9 cards:

1 x Move 1 space1 x Move 2 spaces1 x Move 3 spaces1 x Move 4 spaces2 x Turn any direction3 x Shoot

Approaches to prototyping

how to prototype

Physical Prototypes

1st Person Shooter

Turn sequence

1. Build stack2. Reveal top card3. Resolve Shoot cards4. Resolve turn cards5. Resolve move cards6. Repeat 2-5 for 2nd card7. Repeat 2-5 for 3rd card

Approaches to prototyping

how to prototype

Physical Prototypes

1st Person Shooter

Prototype techniques

how to prototype

Physical Prototypes

Building your own physical prototype

1. Isolate core game mechanicThe actions players repeat most to achieve their goals

2. FoundationDefine game objects (physical setting, resources, units, etc)Determine core game mechanic

3. StructureDefine the turn sequence

4. Formal detailsDefine the rules. Are they all necessary?

5. RefinementPlay test the game

Evaluation Objectives

Evaluation

Obtaining feedback to inform early design concepts

Deciding between different design options

Checking for usability problems

Assessing usability of finished product

As a means for involving people in the design process

Assessing use in practice

Evaluation Step-by-step

1. Establish the aims of the evaluation - the intended users and context of use; obtain or construct scenario

2. Select appropriate evaluation methods

3. Carry out expert review

4. Plan user testing - use the results of the expert review to help focus this

5. Recruit users and organise testing venue and equipment

6. Carry out user testing

7. Analyse results, write up and report back to designers

Evaluation

Play testingRecruiting Play-Testers

Expert evaluationSelf testing is most valuable in the foundation stage of prototyping

User testingFriends and colleagues outside of the design team - but personal relationship can obscure objectivity

Target Audience- Ideal if you can find them- Even better if the play tester is articulate- Should represent the entire spectrum of target audience- Apply NDAs if worried about secrecy

- Continuous batch of new recruits preferable

Evaluation

Play testing

Experts Friends Audience

Foundations

Structure

Formal details

Refinement

Evaluation

Recruiting Play-Testers

Play testingTypical play test schema

1. Welcome play-tester, and thank them for participating

2. Remind play-testers that the game is being evaluated, not the player

- If they find it too easy/hard to play, the game needs improving

3. Ask the play-tester to begin playing when ready

4. Ask them to think aloud during play, and ask questions- Indicate that as an observer, you can not reply to the questions.

5. When play has finished you can interview the play-tester

6. Thank the play-tester. Provide play-tester with incentive.

Evaluation

Play testing

1. Game length

2. Downtime / Interaction

3. Game balance – dominant strategy

4. Analysis paralyses

5. Clarity of rules

6. What do players forget?

7. What do players not use?

8. Which factors increased/decreased player enjoyment?

What to evaluate during a play test session?

Evaluation

Focus groupsStructured group interviews that quickly and inexpensively reveal atarget audience’s desires, experiences, and priorities

(Kuniavsky, 2003)

Good for exploring…desiresmotivationsvaluesfirst-hand experiencesperceptions, not objective truths

Not good for….Obtaining usability informationObtaining quantitative dataProving a hypothesis

Evaluation

Four types of focus groups

Focus groups

1. ExploratoryHow will users understand the productWhat words will people use to talk about the productWhat criteria will people use to judge the product

2. Feature prioritisationWhat features are attractive to the user and why

3. Competitive analysisWhat attracts and repels your audience to alternative products

4. Trend explanationWhat are the user’s motivations and expectations

Conducting a focus groups

Focus groups

1. Set a scheduleA good schedule provides sufficient time for all activities

Timing Activity

T-2wks Determine audience and scope; start recruiting immediately

T-2wks Determine broad topics to be investigated; start writing guide

T-1week Write first version of discussion guide; discuss exact wording

T-3dys Write second version of discussion guide with timing

T-2dys Complete guide; schedule run-through; check equipment

T-1day Run through in morning; check timing and adjust as needed

T Conduct groups

T+3days Watch video tapes; take notes

T+1week Combine notes; write analysis

Conducting a focus groups

2. Pick and recruit target audience

Useful to run several groups with different homogenous audiences

- For example, single sex groups work better

Once identified, write a recruiting profile:

Demographics Game Preferences BehaviourAge: 9-13 Adventure games Buys games monthly Gender: Female RPG

Sims

Evaluation

Conducting a focus groups

3. Define the scope of your research

How many groupings? How many per group?- More than four groups rarely necessary, (three often enough)- 6-8 per group for balance of breadth and depth

Evaluation

Conducting a focus groups

4. Choose the topics you want to researchIdentify 3-5 topics to discuss- i.e., “Discover what factors determine whether they play a particular pinball table.”

Evaluation

Conducting a focus groups

5. Write a discussion guide

Focus group questions should be:- Carefully ordered to make discussion appear natural- Non-directed (not imply an answer or value judgement)

Rather than: “Which is better, X or Y?”Try: “Are there things about X you like? What about Y? Are there ways in which you can compare them?”

A script for the moderator to follow to ensure continuity acrossdifferent groups

Evaluation

Conducting a focus groups

5. Write a discussion guide

Focus group questions should be:- Carefully ordered to make discussion appear natural- Non-directed (not imply an answer or value judgement)- Open ended

A script for the moderator to follow to ensure continuity acrossdifferent groups

- Focused on specifics

Rather than: “Why did you buy Halo 2?”Try: “What factors influenced your decision to buy Halo 2?”

Evaluation

Conducting a focus groups

5. Write a discussion guide

Focus group questions should be:- Carefully ordered to make discussion appear natural- Non-directed (not imply an answer or value judgement)- Open ended

A script for the moderator to follow to ensure continuity acrossdifferent groups

- Focused on specifics- Personal

Rather than: “Which of these features did you like in Halo 2?”

Try: “If you were designing Halo2, which of these features would you include?”

Evaluation

Conducting a focus groups

5. Write a discussion guide

Focus group questions should be:- Carefully ordered to make discussion appear natural- Non-directed (not imply an answer or value judgement)- Open ended

A script for the moderator to follow to ensure continuity acrossdifferent groups

- Focused on specifics- Personal- Unambiguous

See (Kuniavsky, 2003, p216-223) for example discussion guide

Evaluation

Conducting a focus groups

6. Moderating

Moderator guidelines:- Spend some time with the group beforehand- Stick to the guide, but be opportunistic- Engage everyone in discussion

The moderator must balance the needs of the participants’ comfort level and keeping the discussion productive

- Restrict body language- Clarify comments- Restate ideas- Probe for alternative perspectives- Don’t dominate the discussion- Provide time to think

Evaluation

Conducting a focus groups

6. ModeratingThe moderator must balance the needs of the participants’ comfort level and keeping the discussion productive

Asking questions:- Prioritise issues- Write down issues before discussing them- Use participants’ exact thoughts and words, where possible- Key questions need to be asked verbatim- Be ready with examples

Evaluation

Interviews

Interview Structure (1-on-1, or focus groups)

Evaluation

1. Introductions

2. Warm-up

3. General issues

4. Deep focus

5. Retrospective

6. Wrap-up

Interviews

Evaluation

e.g. Rather than: “Which mood board do you prefer? Try: “Is there anything you like about these mood boards?”

Common Problems1. Close-ended questions

Interviews

Evaluation

e.g. Rather than: “Is the strategy guide an important feature for you? Try: “Is there anything about the strategy guide you find interesting?”

Common Problems1. Close-ended questions

2. Questions with complex answers posed as binary questions

Interviews

Evaluation

e.g. Rather than: “What do you do in Zelda when you become hopelessly lost? Try: “When playing Zelda, how do you proceed when you are unsure of the next objective?”

Common Problems1. Close-ended questions

2. Questions with complex answers posed as binary questions

3. Loaded words, or words with multiple meanings

Interviews

Common Problems

Evaluation

1. Close-ended questions

2. Questions with complex answers posed as binary questions

3. Loaded words, or words with multiple meanings

4. Asking people the future

5. Invocation of authority or peer pressure

e.g. “The designer has a lot of experience developing game characters. What did you think of the lead character.” “People generally find the fairy navigator confusing. What was your experience?”

Interviews

Common Problems

Evaluation

1. Close-ended questions

2. Questions with complex answers posed as binary questions

3. Loaded words, or words with multiple meanings

4. Asking people the future

5. Invocation of authority or peer pressure

6. Assuming you know the answer

7. Assuming they can answer the question

Questionnaires

Evaluation

A set of questions that creates a structured way of asking a large groupof people to describe themselves, their interests, and their preferences

Question Categories1. Characteristic categories [who they are, what consoles do they have] - demographic

2. Behavioural categories [how they behave] - purchasing habits (how often do you buy games?) - game genres most played

3. Attitudinal Categories [what they want and believe]- satisfaction (do they like your product?, meet their requirements?) - preference (which features do they like most?) - desire (what features is the game lacking?)

Questionnaires

Evaluation

A set of questions that creates a structured way of asking a large groupof people to describe themselves, their interests, and their preferences

QuestionsUse open-ended questions sparingly.

Close-ended questions should be…

- specific

- exhaustive

- mutually exclusive

Questionnaires

Evaluation

A set of questions that creates a structured way of asking a large groupof people to describe themselves, their interests, and their preferences

QuestionsUse open-ended questions sparingly.

Better to use…

Multiple choice questions

Q10. How long have you owned your PlayStation 3?

O less than one yearO 1 – 2 yearsO 2 years or more

Questionnaires

Evaluation

A set of questions that creates a structured way of asking a large groupof people to describe themselves, their interests, and their preferences

QuestionsUse open-ended questions sparingly.

Better to use…

Checkboxes

Q9. Which consoles do you own?

[ ] Sony PlayStation 3[ ] Microsoft XBOX 360[ ] Wii U

Questionnaires

Evaluation

A set of questions that creates a structured way of asking a large groupof people to describe themselves, their interests, and their preferences

QuestionsUse open-ended questions sparingly.

Better to use…

Likert scales

Q9. Rate the following aspects of XYZ Pinball

Very Good | Good | Neutral | Poor | Very PoorGraphics O O O O O

Questionnaires

Evaluation

Sampling size

The number of surveys depend on the variation in your population- greater the variation in your population, the larger the sample needed

How to determine sample size

1. Estimate size of population being studied

2. Apply the following table

Population Sample Size

100010,000100,000

150300800

Ethnographic

Evaluation

Structured observation

Behaviour categories defined before observation

Training of observers to ensure inter-observer reliability

Consistency in sampling method

Data Gathering Devices

Film/video, still camera, audio tape, notes, coding system

Data Gathering Systems

Rating Behaviour (1-10)

Coding Behaviour (GO: Gives Object)

Ethnographic

Evaluation

Behaviour categories defined before observation

Training of observers to ensure inter-observer reliability

Consistency in sampling method

Data Gathering Devices

Film/video, still camera, audio tape, notes, coding system

Data Gathering Systems

Rating Behaviour (1-10)

Coding Behaviour (GO: Gives Object)

Structured observation

Ethnographic

Evaluation

Participant observation

Full participant

Participant as observer

Observer’s role is hidden, and they become an authentic member of the group

Observer’s role is not hidden, but it is not considered to bethe main purpose of participation

Observer as participant

Main role is as observer, and accepted into group as such

Conclusions

Benefit of prototyping

Multiple evaluation tools, including:

- Play Testing- Focus Groups- Interviews- Questionnaires- Ethnographic Studies