game accessibility at hanze hogeschool

Human Computer Interaction Lab

University of Nevada, Reno

Playing Games without Visual Feedback

Eelke Folmer, Associate Professor

Human Computer Interaction Lab


reno

Player-Game Interaction Research


Research Interests

Accessible Computing Visual Impairments Spatial Perception Video Games Augmented Reality 3D interfaces Wearable computing Natural Interfaces Sonification Haptics Human Navigation CrowdSourcing Sensory Substitution exergaming Spatial computing

Automotive Interfaces Virtual Reality Gestures Mobile computing



Overview of Talk

What are some of the barriers that blind people face when playing video games?

Why should video games be accessible to users who are blind?

How can virtual world & video games be made accessible?



How do we play games?

Abstraction

1. feedback2. fire gun3. pull trigger

1. feedback2. kick & punch3. move feet & arm

1. feedback2. aim for point3. drag with finger



Interaction Model



Gaming with Visual Impairment

retintis pigmentosa color blindness

macular degeneration blindness



Interaction Model



Sensory Substitution Research

Visual

AudioHaptic

information

closed captioning

braille

speech synthesis



Research Challenges

Visual

AudioHaptic

information

informationinformation

Why should people with Visual Impairments be able to play video games?



Benefits

Social

HealthEmployment

Education



Visual Impairment

1.3 million people who are blind (60k kids) [US]6.8 million who have a visual impairment [US]Population of VI expected to double [WHO]Obesity --> cataracts --> blindness



“Extreme Interaction” Design Design Interfaces for “extreme” playersGames are complex real time simulationsDisability as a driver of innovation

speech synthesis

screen readers

speech recognition

automatic captions



Innovation by solving extreme problems

? ?

Human Computer Interaction Research


Example accessible interfaces

Natural Language Virtual World Interfaces»Text SL »Syntherella

Non-Visual Natural User Interfaces (Exergames)»VI Tennis»VI Bowling

»Seek-n-tag

»Pet-n-Punch»RTSS

Natural Language Virtual World Interface

HCC-Small: TextSL: A Virtual World Interface for Visually Impaired, National Science Foundation Eelke Folmer (PI), George Bebis (Co-Pi). Amount: $499,332. SGER: Developing an Accessible Client for Second Life, National Science Foundation Eelke Folmer (PI), Amount: $90,488.

TextSL

Player Game Interaction Research


Virtual WorldsEducation Museums Communities



Web Accessibility

“ The eiffel tower was built in ...”“alttag:picture of the eiffel tower”

Screen reader

text/images



Virtual World Accessibility

“.............................”

Screen reader

pixels



Inspiration

Command Line Interface»Text only»Iterative »Natural language»Screen reader accessible

Zork

web based interface»screenreader /

speechbasic commands

» navigation

» exploration

Text SL

» communication

» interaction



Natural Language Interpreter

Avoid learning specific commandsIntuitive & Easy to Learn

walk

go

move

move

natural language prepositions

move

tochai

rthe

adjectives

give my

flower to jane



SL Content Problems

Densely populated with objects »Overwhelm users with feedback»Difficult to navigate collision free

Lack of meta data (40%)»underwhelm users with feedback

tree

?fire

bike

dogtree

moe

jack

this object has a really long name

wall

chair

tree

table

jill

curly

carchest

larrybike bike

bike

tree

tree

?

?

?

?

?

?

bike

car

car



Navigation

Basic Obstacle avoidanceTeleport when stuckMake avatar appear as normal as possible

>move north 10>move to name (object / avatar)>teleport to Help Island



Communication>say “Hello” >whisper to Jack “bla”>mute Jack

WTF?

>sit on chair



Interaction>touch billboard

- no textual output (yet)



Iterative query mechanisms

chair

fire

dog

jill

>describe“you see avatar jill, and the objects chair, a fire and a dog”>describe dog“the dog is a golden retriever and is for Sale for $10”>where dog“the dog is 10 meters to your right”



Syntherella: Scene Synthesizer

tree

fire

bike

dogtree

jack

wall

chair

tree

table

curly

carchest

larrybike

bike

tree

?

?

?? bike

object with a really long name

moejill

bike

tree

?

?

?

car

tree

larrybike?bike

dog

dog

dog

chest

chest

- Content within range - Remove non descriptive objects

- Based on #worldlimitif #words > #wl {abstraction}else {detailing}



Abstraction

>describe“there are 3 animals and 3 vehicles

bikecatdogtruc

kcarpig

vehiclesanimals

object taxonomy

>describe“there is a cat a dog, a car, a bike a car, a truck and a pig

bike

cat

dog

truck

car

car

pig



Detailing

>describe“there is a cat and a dog and a car

>describe“to your left is a black cat and a brown dog who is barking. In front of you is a red race car.



User Studies

slower

slower

slower

same



Lack of Meta data

?

?

?

?

?

?

?

>describe“there is an object an object an object an object an object etc...



Labeling images vs 3D objects

- Defined in Isolation- Solid bodies (prims)- Efficient discrimination

- Segmentation- 2D info



Construct Classifier

object categories

shape descriptors

unknown object



how to derive object categories?

dog?



Use Human Computation

find a cat

label

label

label



Seek-n-TagGoal:

»Player participates in Scavenger hunt

Rules:»30 seconds to “tag” an object»Score 50 points per tagged object»Initially start with 900 seconds»Game over when time =0»Remaining time is added to clock.

Competition»Leaderboard

Find a Cat

Find a Tree

Find a Dog

Find a Shoe

✔ + 5s

✔ + 15s

✘ - 30s

✔ + 10s



User study

Manual labeling versus labeling with a game

Seek-n-tag faster Seek-n-tag takes fewer labeling attempts

to reach consensus on object name.

object

object objec

t

object

object

object

object

object objec

t



Manual labeling using AMT

90%

1000

500

Currently Researching



Interaction with interactive Objects

>touch billboard>touch horse

Collect descriptions using AMTSemantic analysis of descriptions to understand

what properties of objects play a roleSee if properties can be parsed from state

changes in object

>create green cube “block1”



Content Creation

>create green cube “block1”>link block1 block2

>create brown d0g



Publications / More info

ear.textsl.org

• Bugra Oktay, Eelke Folmer. Syntherella: A Feedback Synthesizer for Efficient Exploration of Virtual Worlds using a Screen Reader,Proceedings of Graphics Interface 2011. Pages 65-70, St John, Canada, May 2011.

• Bei Yuan, Manjari Sapre, Eelke Folmer. Seek-n-Tag: A Game for Labeling and Classifying Virtual World Objects, Proceedings of Graphics Interface (GI). Pages 201-208, Ottawa, Ontario, June 2010.

• Eelke Folmer, Bei Yuan, Dave Carr, Manjari Sapre. TextSL: A Command-Based Virtual World Interface for the Visually Impaired, In Proceedings of the 11th international ACM SIGACCESS conference on Computers and Accessibility, Pages 59-66, Pittsburgh, Pennsylvania, October 2009.

code.google.com/p/textsl

Try ?

Contribute?

http://www.textsl.org/

http://portal.acm.org/citation.cfm?id=1992928

http://portal.acm.org/citation.cfm?id=1992928

http://portal.acm.org/citation.cfm?id=1839214.1839249&coll=ACM&dl=ACM&CFID=103925613&CFTOKEN=54617906

http://portal.acm.org/citation.cfm?id=1839214.1839249&coll=ACM&dl=ACM&CFID=103925613&CFTOKEN=54617906

http://doi.acm.org/10.1145/1639642.1639654


Non Visual Natural User Interfaces

HCC: Small: Proprioceptive Displays to Engage Blind Users into Healthy Whole Body Interaction, National Science Foundation Eelke Folmer (PI), Amount: $420,320.

John Folycollaborators:Lauren Liebermann

VI Fit



Gesture Based Interaction



Reliance on Visual Cues

dodge

♫♫



visual impairment & obesity

60,000 Blind Children in US More obese



Barriers to physical activity

Rely on sighted guide Safety



Exercise games

★ Moderate to Vigorous PA (MVPA)★60 min of MVPA daily (CDC)

★Independently★Safer



Research question



Gestures w/o visual cues?

?



Physical Activity = Spatial-Temporalwhere

?

when?

Target Acquisition



Exergames are temporal-spatial

Kinect Sports Eye Toy Kinetic

★Temporal/Spatial: Jump when hurlde is close

★ Spatial: Punch/ Kick Targets★ Temporal: Dodge target



Sensory Substitution Research

Visual

AudioHaptic

information

target acquisition ??(directed) gestures



Constraints with regard to SS

Social Contexts Music based

cool!!

Audio Haptic?



Temporal Challenge

serve

Wii Sports

Buzz

♫

♫bounce

return

VI Tennis



Sensory Substitution

VI Tennis

WII Tennis



Wii Bowling VI Bowling

Spatial Challenge

VI Bowling

Wii Bowling



Studies @ Camp Abilities

Vi Tennis/Bowling yield Active energy expenditure that is considered moderate physical activity

Kids need 20 minutes of moderate to vigorous PA



User studies

light

moderate

vigorous

VI Tennis

Upper Body

Whole Body



Pet-n-Punch



Instrumentation



Real time Video Analysis<SECTION> <XMIN>100</XMIN> <XMAX>300</XMAX> <YMIN>150</YMIN> <YMAX>300</YMAX> <MINR>230</MINR> <MAXR>999</MAXR> <MING>230</MING> <MAXG>999</MAXG> <MINB>0</MINB> <MAXB>100</MAXB> <ACTION <WIIMOTE> <ID>0</ID> <DURATION>-1</DURATION> </WIIMOTE> <AUDIO> <ID>alert.wav</ID> <LOOP>0</LOOP> </AUDIO> </ACTION></SECTION>

QuickTime™ and a decompressor

are needed to see this picture.



VI Ski

Related Projects



Publications / More info

vifit.orgcode.google.com/p/vifit

Try ?

Contribute?

•Tony Morelli, Eelke Folmer. Real-time Sensory Substitution to Enable Players who are Blind to Play Gesture based Videogames, In Proceedings of Foundations of Digital Interactive Games, to appear, Bordeaux France, June 2011•Tony Morelli, John Foley, Lauren Lieberman, Eelke Folmer. Pet-N-Punch: Upper Body Tactile/Audio Exergame to Engage Children with Visual Impairments into Physical Activity, Proceedings of Graphics Interface (GI) •Tony Morelli, John Foley, Eelke Folmer VI Bowling: A Tactile Spatial Exergame for Individuals with Visual Impairments , In Proceedings of ASSETS, Pages 179-186, Orlando, Florida, October 2010. •Tony Morelli, John Foley, Luis Columna, Lauren Lieberman, Eelke Folmer. VI-Tennis: a Vibrotactile/Audio Exergame for Players who are Visually Impaired, Proceedings of FDG, Pages 147-154, Monterey, California, June 2011



http://www.eelke.com/files/pubs/rtss.pdf

http://www.eelke.com/files/pubs/rtss.pdf

http://vifit.org/files/pubs/petnpunch.pdf




http://portal.acm.org/citation.cfm?id=1878836&CFID=111766415&CFTOKEN=36853823

http://portal.acm.org/citation.cfm?id=1878836&CFID=111766415&CFTOKEN=36853823

game accessibility at hanze hogeschool

Education

content creators

analyzed large

game interaction

compile meaningful

gesture based

called sensory

representing

visual impairments