A Practical Guide to Doing Ethical Player Testing Mia Consalvo Concordia University

Who am I? ●  Studying game players since 2000 ●  Use a variety of methods to better

understand play ●  Wrote Cheating, Players and their Pets,

Atari to Zelda ●  Run the mLab at Concordia University

Outline for today’s talk ●  A brief history of why ethical research became a

thing at universities ●  Important concepts to keep in mind for testing ●  Questions to ask yourself when prepping for

testing ●  What about VR? ●  Q&A

What this talk is NOT about ●  Improving your game design through

player testing

What this talk IS about ●  Ensuring play testers are getting the best

possible experience (and want to come back again)

●  Improving the quality of testing feedback

A caveat ●  “Following a code of ethics is not the

same as being ethical. A domain-specific ethics code … can never function as a substitute for ethical reasoning itself.”

A brief history of human subjects research

The Milgram experiment (1961) ●  Studied obedience to authority figures via

willingness to give increasingly severe ‘shocks’ to others

●  65% of participants administered final massive 450-volt shock

●  Study criticized for use of deception, other flaws

Stanford Prison Experiment (1971) ●  A study of the psychological effects of

becoming a prisoner or prison guard ●  Conducted by Philip Zimbardo with a

group of college students ●  Prisoners ‘arrested’ at their homes, driven

to prison, processed, given jumpsuit and assigned an ID number

Stanford Prison Experiment ●  Experiment abandoned after six days –

some prisoners had breakdowns ●  Subjects could not leave voluntarily ●  Zimbardo was not a neutral observer ●  No debriefing was done to assess

potential harms done

The introduction of protocols and guidelines for doing federally funded research

Consent

What is consent? ●  Free, informed and ongoing consent ●  Free = voluntary

Voluntary Consent ●  No Undue Influence or Coercion ●  Is there a power relationship between the

Investigator and potential participants? ●  Is the investigator relying on the trust or

dependency of particular people (ex: my students)?

●  Threat of harm or punishment for failure to participate?

Informed Consent

●  Participants know what is expected of them during the test

●  How long the test will last ●  How will they be observed/recorded ●  It’s okay to change their mind

Informed Consent ●  Told of foreseeable risks and potential

benefits ●  Under no obligation to participate, free to

withdraw at any time ●  Whether participants will be identified

directly or indirectly

Informed Consent ●  Contact information if they have

questions after the fact ●  Information about how data will be used,

who it (may be/will be) disclosed to ●  Information about payments or incentives ●  Time to consider all of this, ask questions

Ongoing Consent ●  Consent is not a ‘once and you’re good’

decision ●  Consent can be negotiated if there are

different aspects to the play test

Withdrawing Consent ●  Testers should be told they can stop the

test at any time ●  The environment should support this

possibility ●  There should be multiple ways to

withdraw (verbal, nonverbal/written)

Using incentives to participate ●  Incentives can attract more potential

participants ●  If incentives are very large or valuable this

may encourage participants to disregard risks ●  Economic circumstances of participant pool,

age and capacity, customs and practices

Use of deception ●  Deception can be a part of ethical

research ●  What is the risk involved in the

deception? ●  Does the benefit of deception outweigh

potential risks?

Harm and Risk

Defining Harm and Risk ●  Minimal risk: probability and magnitude

of possible harms implied by participation in the study is no greater than those encountered by participants in those aspects of their everyday life that relate to the research

Beyond minimal risk

Vulnerable Populations ●  Children ●  Diminished capacity ●  Indigenous peoples

Children ●  Consent usually obtained through parents/

guardians ●  Consider the age range you are including and

why ●  Children who are 7, 10 and 13 are very different

in terms of what they understand as risks, as feedback, as consent

Children ●  During testing, nonverbal communication

can be more important than what is being said (for both testers and researchers)

Children ●  For a detailed examination of kids and

play testing, check out Gareth Griffiths’ 2014 GDC talk “Child’s Play: Playtesting with Children in the World of Skylanders”

●  http://www.gdcvault.com/play/1020348/Child-s-Play-Playtesting-with

Privacy and Confidentiality

Privacy and Confidentiality ●  How are you collecting data? ●  How are you storing data? ●  How are participants identified? ●  What are you doing with the data? ●  What happens to the data when the study

is over?

Privacy and Confidentiality ●  Levels of disclosure of participant identity

●  Anonymous ●  Pseudonyms/Confidential ●  Identifiable

●  What personal information are you collecting and how might it be linked to participants’ tests?

Special Considerations for VR

“VR poses risks that are novel, that go beyond the risks of traditional psychological experiments in isolated environments & go beyond risks of existing media technology for the general public” –Madary & Metzinger, 2016

“the virtual pit” ●  Subject with HMD stands on ‘ledge’ and asked to

lean over and drop a beanbag into a deep pit ●  Subject stands on wooden platform 1.5” from

the ground ●  Showed increased signs of stress through

increases in heart rate, skin conductance

Remember the Milgram Experiment? ●  Re-done in VR environment ●  Asked participants to administer shocks

to a virtual human performing memory tests [they knew she was virtual]

--Slater M, Antley A, Davison A, Swapp D, Guger C, et al. (2006) A Virtual Reprise of the Stanley Milgram Obedience Experiments. PLoS ONE 1(1): e39. doi:10.1371/journal.pone.0000039

Virtual Reprise ●  2 participants emphasized correct answers while

reading words ‘in an attempt to help her’ ●  8 repeated the question after receiving no

response ●  Voices of some participants showed increasing

frustration with wrong answers

Virtual Reprise ●  Often behaved in a way that only made sense if

they were responding to the virtual character as if she were real

●  Humans respond realistically at subjective, physiological, and behavioral levels in interactions with virtual characters notwithstanding their cognitive certainty that they are not real

Virtual Reprise ●  Might be continued evidence that subjects will

be obedient to an authority figure ●  May be a matter of participants being willing to

put up with their own discomfort for the sake of honoring their agreement to be a participant in the experiment

Virtual Reprise ●  Might be continued evidence that subjects will

be obedient to an authority figure ●  May be a matter of participants being willing to

put up with their own discomfort for the sake of honoring their agreement to be a participant in the experiment

Proteus Effect ●  Subjects conform to the behavior that

they believe others would expect them to have based on the appearance of their avatar

●  Behavior in VE can have lasting psychological impact after subjects return to the physical world

Guidelines for VR research ●  Do no harm ●  No real history we can use as a source for

insight ●  Tautology?

Guidelines for VR Research ●  Have an explicit statement (or explain) to

the effect that “immersive VR can have lasting behavioral influences on subjects, and some of these risks may presently be unknown”

Guidelines for VR Research ●  Tell people that they may have powerful

emotional responses to game content whether or not they ‘believe’ it’s real

●  Torture in a virtual environment “is still torture”

Guidelines for VR Research ●  Many new, additional kinds of data being

collected ●  Eye-movements, emotions, real-time reactions, bodily

movements (mo-cap) ●  One’s kinematics may be uniquely connected to one’s

identity ●  Consider what data you really need to keep and

what might be erased to preserve privacy

Broader considerations beyond testing

●  We don’t know the psychological impact of long term immersion in VR (addiction, manipulation of agency, unnoticed psychological change, mental illness)

●  The potential for abuse of avatars that look like their users

Creating a protocol for play testing – Questions to guide you

What am I testing for? ●  Do players understand what to do in the

first level? ●  Do my instructions make sense or are

players floundering around? ●  How long do players take to get to the

end of the level?

Recruitment ●  Who is my player base? How can I recruit

players from that group without exploiting trust/power relationship?

●  Should I provide an incentive or reward for play testing?

●  Where am I asking them to play test?

Informed Consent ●  How will I handle obtaining consent? ●  What do I say about withdrawing

consent? ●  How can I ensure testers feel comfortable

enough to leave if they really want to quit testing?

Deception ●  Is there anything in the game I want to be a

surprise to testers? ●  Is there a way to let testers know there’s some

material they might be really bothered by without spoiling the content for everyone?

●  Is this ‘surprise’ really worth it?

Harm and Risk – VR Edition ●  Having a ‘kill switch’ ●  Testers’ avatars have protective bubble ●  Protected populations issues

●  Find information or agencies that testers might want to contact if they are bothered by content in your game

Privacy and Confidentiality ●  What should I record – gameplay, tester audio,

tester video, physiological responses? ●  If I share data with anyone, am I keeping tester

identities anonymous or confidential? ●  How am I storing data and keeping it secure? ●  What am I doing with that data after testing is

done?

Questions for multiplayer testing ●  Do players understand how they can interact

with other players? ●  Do instructions for that make sense? ●  Are players using anti-harassment tools we

created? ●  What sorts of toxicity (if any) am I seeing during

play?

Harm and risk – an aside ●  Be wary of defining what ‘harassment’

looks like/sounds like ●  Ask what they might have seen done/said

about other players, not just this player ●  ‘Abuse’ versus ‘Drama’

Putting it all together – Consent Form Language/Guidelines for testers to understand and agree to

Example Consent Form Text ●  “I understand that I have been asked to

participate in play testing of Eksa: Isle of the Wisekind by Mia Consalvo of GAMBIT Studios, contact information HERE.”

Example Consent Form Text ●  “I have been informed that the purpose

of the play testing is to determine [how clear directions are in the game, and how challenging puzzles are in the first few levels.]”

Example Consent Form Text ●  “I understand that I am being asked to

play a game that isn’t finished, for about 30 minutes.”

●  “I understand that the computer is video recording my gameplay and I am being observed by someone while I play, who is taking written notes.”

Example Consent Form Text ●  “I understand that I am free to withdraw

my consent and discontinue my participation at any time without negative consequences.”

Example Consent Form Text ●  “There might be certain risks in

participating in this play test. These risks include feeling frustration or anger while playing, or hearing other players saying negative things during gameplay which might upset me.”

Example Consent Form Text ●  “I understand that my participation in this

study is… ●  CONFIDENTIAL (the researcher will know, but

will not disclose my identity) ●  NON-CONFIDENTIAL (my identity will be

revealed in study results)”

Example Consent Form Text ●  “I have carefully studied the above and

understand this agreement. I freely consent and voluntarily agree to participate in this study.”

Additional Resources ●  For US-based developers

●  https://phrp.nihtraining.com/index.php ●  The Belmont Report ●  http://www.hhs.gov/ohrp/regulations-and-policy/belmont-

report/index.html

Additional Resources ●  For Canadian-based developers

●  http://www.pre.ethics.gc.ca/eng/policy-politique/initiatives/tcps2-eptc2/Default/

●  For European-based developers ●  http://www.eurecnet.org/index.html

Additional Resources ●  Madary M and Metzinger TK (2016) Real

Virtuality: A Code of Ethical Conduct. Recommendations for Good Scientific Practice and the Consumers of VR-Technology. Frontiers in Robotics and AI. 3:3. doi: 10.3389/frobt.2016.00003

Questions?