exploring eye tracking for games user research: a case study of lessons learned

Exploring Eye Tracking for Games User Research! A case study of lessons learned

Ania Rodriguez, M.S.CEO/FounderKey Lime Interactive@KeyLimeInteract

Karl Steiner, Ph.D.Manager of UsabilityTHQ, Inc@THQ

© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI

2

Overview

Introductions

Player Research Goals, Techniques and Challenges

Think Aloud vs. Debrief

Our Game Plan

Steps to Success

Q&A

© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010

3

About Key Lime Interactive Experienced market research and usability

professionals Founded January 2009 Expertise in both Eye-Tracking in-lab studies as

well as remote usability testing techniques

Cross-Industry Expertise Ability to scorecard against competition Ability to suggest cross-industry

recommendations to improve ease of use

Independent Third Party Impartial with no-hidden agenda Ability to work with business team and

developers to “get everyone onboard”


Clients:

7/26/2010

4

About THQ

THQ Inc. (NASDAQ: THQI) is a leading worldwide developer and publisher of interactive entertainment software. The company develops its products for all popular game systems, personal computers and wireless devices. Headquartered in Los Angeles County, California, THQ sells product through its global network of offices located in North America, Europe and Asia Pacific.


5

What Makes Software Usable?

Speed?

Efficiency?

Simplicity?

Intuitiveness?

6

A “Usable” Game

Click Here to WIN!!!

7

Usability ≠ Playability

For games, the user experience is the primary purpose of the product Games must not only be usable… But also enjoyable

Part of the appeal of (many) games is overcoming a level of challenge Games should be difficult… But not too difficult

Games employ a variety of devices, used by a variety of players, in a variety of physical settings Despite common stereotypes, gamers represent an

increasingly diverse group Gameplay styles and techniques that are

appropriate for one audience in one setting playing one type of game may not be acceptable in other contexts


8

Player Research Goals & Techniques Player Research Goals

Describe and understand common players behaviors within a game

Determine where and why players encounters issues or difficulties when playing a game

Assess when and how players enjoy the experience of playing the game

Player Research Techniques Qualitative Techniques

Lab-based “usability” style tests Quantitative Techniques

Surveys and logs of player interactions


9

Using Observation and “Think Aloud” to diagnose player difficulties in a 3D Game

We can observe if player is having difficulties Fails a task Takes too long / too many attempts Gets visibly frustrated

We require additional insights to determine why player is having difficulties May observe player taking incorrect actions or using incorrect

controls Player may verbally describe incorrect assumptions about their

goals or incorrect plans for achieving objectives (Think Aloud or Post-Session Debrief)


10

Anatomy of a 3D Game Screen


11

Objects in 3D World

EnemiesEnemiesEnemies

BuildingsEnemiesBuildingsTerrain


12

Heads-Up Display (HUD)

Score / Timer

Target Reticle

Mini-mapTarget ReticlePlayer Status


13

Issues With Think Aloud for Player Research in 3D Games

Many gameplay issues are related to player awareness and understanding of in-game cues and objects

However, player observation and think-aloud techniques do not always provide adequate information regarding player awareness and understanding of these elements

Players may notice details but fail to understand or act on the information (making observation difficult)

Players may not mention details they noticed while they play

Players may not remember details if they are queried after play


14

Our Game Plan

Take a 3D game that has already had been tested with Think Aloud Protocol and rerun the study with Eye-Tracking

Gather additional insights into player gaze and awareness, such as: What objects in the 3D world are noticed? When are they noticed? Do

players comment on or remember these objects? What objects in the 3D world are missed? What elements in the HUD are noticed? At what points are they noticed?

Do players comment on or remember these HUD elements? What elements in the HUD are missed? Do overall gaze patterns change in different areas of the game?


How we did itLessons learned from running our first eye-tracking study in Gaming


16

Step#1: Computing Power

You need to network the eye-tracker to a separate PC or gaming console.

Pre-release games may not have been tuned for performance, so high-end hardware is typically required. If testing on a PC, consider: CPU – A high-end recent CPU (e.g. Intel Core 2 Quad) Video Card – Dedicated video card (laptops with integrated video or low-

performance cards will likely not be sufficient) Video output – make sure that your hardware supports the necessary game

output (e.g. HDMI, VGA, etc.) and is compatible with the eye-tracker hardware

RAM – 4GB (8GB is always better)

Our gaming computer: Gateway FX6831-01 PT.GAH02.004 Desktop PC - Intel Core i7-860 2.8GHz,

8GB DDR3, 1.5TB HDD, DVDRW, ATI Radeon HD5850, HDMI, Display Port, eSATA, FireWire, Windows 7 Home Premium


17

Monitor #1 with

SMI EyeTracker

XboxController

Monitor #2 with

SMI ExperimentCenter Software

Laptop runningSMI iView X™ Software

The Gaming Machine

Step#2: Double PC Setup


18

Step#3: Testing Protocol

In a game study, some players may not volunteer much information using just a Think Aloud.

The game itself may demand their attention at times, and describing the experience can introduce additional cognitive load or interruptions to game play

Our test scenarios included a natural stopping point midway through the study

Our suggestion: Use ‘debrief’ rather than Think Aloud because: Ability to observe what the participants would naturally do

without disrupting how they would play if they were on their own Adding Think Aloud to this experience adds more cognitive load

to the task than normally would be present


19

Step#4: Clarify Goals and Terminology

Clearly stated test goals and plans are important for any domain

Fleeting game events and unique game terminology makes prior planning and preparation even more important

Our suggestion: review test goals with project sponsors in front of the game Game events may appear quickly – observers need to know what to

look for and at what time Game terminology can often be arcane and very specific. For

example, an observer may need to know the visual differences between: Space Marines and Chaos Marines M-16s and AK-47s Atomic Drops and a Facebreaker DDTs


20

Step #5: Run Study

8 gamers with third-person shooter experience 7 males, 1 female

1 day of testing; 60 minute sessions

30 minutes between subjects

Session overview: Scene 1 (10-15 mins) Debrief (5-10 mins) Scene 2 (10-15 mins) Debrief (5-10 mins)

One participant, 1 moderator

2 Observers/Note Takers


21

Step #5: Analysis

Fixations are generally an inadequate way of measuring subject attention when viewing dynamic media. This is because a subject, when tracking a moving target (like a

soccer ball), is actually performing smooth pursuit. This is similar to fixation, at least cognitively, but it is not detectable via normal fixation detection algorithms.

Instead, we recommend dwell time. This is the summary of time spent within an area whether in fixation, saccade, or smooth pursuit, and is a better way to quantify attention on a moving object.

Glance time is a related and equally useful measure.


22

Definitions

Dwell Time: Dwell time starts at the moment the AOI is fixated and ends at the moment the last fixation on the AOI ends(equal to the sum of durations from all fixations and saccades that hit the AOI)

Glance Duration = Saccade duration for entering the object + sum of all fixation durations and saccade durations before the eyes begin to leave the AOI(equal to the dwell time + duration of saccade entering AOI)

7/26/2010© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI

23

Step #6: Gridded AOIs [Dwell Time]


After 30 seconds End 60 seconds

Example: Gridded Areas of Interests (AOIs) for E3 Promotional Video

7/26/2010

24

Step #7: AOIs and KPIs


Create AOIs

• AOIs can be • Any shape or form• Made invisible (turn on/off)• Enabled/Disabled

7/26/2010

25

Step #8: Example of Dwell time on Custom AOIs


Pick object(e.g. skull)

7/26/2010

26

Step #9: Create Report

Create typical usability report with findings Executive Summary Methodology Findings and Recommendations Detailed Findings and Metrics Appendix

Present Include everyone you can (e.g., usability team, senior

leadership, developers, project managers, etc.)


27

Step #10: Highlight Videos


Show Sample VideoEye-tracking playback of the online publically available promotional video on THQ’s site

7/26/2010


Ania Rodriguez, M.S. Key Lime Interactive Founder [email protected] @keylimeinteract

Karl Steiner, Ph.D. Manger of Usability Research

[email protected]

mailto:[email protected]

http://email03.secureserver.net/search.php

exploring eye tracking for games user research: a case study of lessons learned

Self Improvement