modularity and multimodal interaction brian fisher …fisher/siggraph2002coursenotesbf.pdf5 fels:...

1

THE UNIVERSITYTHE UNIVERSITYOF BRITISH COLUMBIAOF BRITISH COLUMBIA

Intersensory Interactions and HumanComputer Interaction (HCI)

Brian Fisher

Fels: Design of Interactive Multimodal Graphics

Intersensory Interactions

• Intro and metacognitive gap

• Integrating cogsci theory with design

• Cognitive Architecture

– Modularity and multimodal interaction

• Information hiding-- conflict resolution

• Cognitive impenetrability

• Performance differences between modules

• Recalibration

– Spatial indexes in complex environments• Multimodal cue matching within modules


Key Points for HCI Practice

• The Metacognitive Gap: The need for agrounded Cognitive Science approach

• Reflective design practice methods– Integrating CogSci with interaction design– Iterative design cycle

• Examples of extended HCI– Cognitive architecture: Multimodal displays and

how they are understood– Situated cognition: embodied interaction with

complex displays


Introduction

History -- why we need to adapt HCImethods for multimodal interaction

2


HCI History

• “Classic” HCI: User/Task/Tool Model– Task, Protocol & GOMS/keystroke analysis

• Command-line, menus, workplace systems

• Theoretical underpinnings– Cogsci of conscious thought

– Learning, Memory, Reasoning

– Sequences of operations


Ergonomic HCI domain limits

TaskTool

User

What if I am doing this for fun?What if I want new insights?What if I want to communicate with someone?What if I am exploring a complex environment?


Evolution towards multimodal displays

• WIMP interface: visual semantics– Metaphorical tool icons on desktop

– Direct manipulation

• Information visualization: Informationprocessing in the visual system– Visual analogs of information

– Spatial Instruments


Iterative development cycle

Walkthrough or experiment

Implement prototype

DesignDesign TestTest

Spiral with increasing detail and test specificity

How? What?

3


Future challenges

• Perception, cognition, and action in an immersive

multimodal environment populated with objects

events and actors

• Applications in entertainment, cognition,

communication

• Blend of virtual and real spaces… with seams

– Are the rules consistent?

– Can users shift between them?

– Can frames support rule shifts?Fels: Design of Interactive Multimodal Graphics

Opportunities for creative design

• Environments: Affordances for exploration– Spatial cognition, human space constancy theory

• Support for creative & logical thinking– Problem solving, embodied cognition models

• Media-based communication & collaboration– Metacognition, distributed cognition

• Experience (Kansei) engineering: Movingbeyond usability


Effective Interface Design for RichSensory Environments

The interaction between display characteristics andthe information processing characteristics of the

user’s perceptual, motor, and cognitive processeswill largely determine interface performance


Metacognitive Gap

• Intuitions about thoughts,goals and plans (“folkPsychology”) are reasonably accurate

• Intuitions about how people see, hear, andremember are very inaccurate

• Lack of awareness of the limits of intuition isthe “Metacognitive gap”

• Design-by-intuition leads to bad userexperience

4


Evolving interaction design models

• Guidelines are (still) inadequate• User-centred design inadequate for

rich sensory environments• Need for theory-rich, evidence-based

approach: design for lower-levelprocesses

• Must integrate with higher-ordercognitive task analysis and user-centred design practices


Bridging the theory/practice gap

• Convincing designers that there is something to

understand—the “metacognitive gap” of folk

Psychology

• Convincing Cognitive Scientists to answer relevant

questions—Complex data displays and multiple

tasks and the Psychophysics reductionist approach

• Integrating research and design—Finding a

common language


Reflective HCI Practice (Schön)

Test: walkthrough, field study or experiment

Exp:“Lab sense”,FS: Observation

Ideas and hypotheses

Evaluation, Mapping

Implement

Interaction Design

Information foraging

(Online) Literature:PsychologyKinesiologySociologyAnthropologyArchitecture…


Fields of interest: ExperimentalPsychology:

– Founded~ 100 years ago

– Areas of study• Psychophysics—Vision, hearing, tactile

senses

• Attention—Endogenous, exogenous,sustained

• Learning and memory

• Goal is often information processingalgorithms

• Discussed in Module 2

5


Useful things from Psychology

• Perception and attention– Visual & auditory acuity & discrimination– Colour perception– Salient display changes, change blindness

• Learning and memory– Primacy, recency– Skill acquisition– STM limits– State-dependent learning

• Decision making– Base-rate neglect– End effects, biases


Fields of interest: Kinesiology &related disciplines

• Science of human movement– Neuroscience

– Mechanics

– Anthropometry

• Examples: Fitts’ law, GOMS/Keystroke

• Goal is often perceptually guided behaviour

• Discussed in Module 3


Fields of interest: Cognitive Science

• Cogsci Society founded 22 years ago• Combines Experimental Psychology,

AI, Philosophy and Neurophysiology• 3 levels of analysis

– Semantics: Intentions, Goals, andMeanings

– Syntax: Information processing– Implementation: Neural processing

• Goal is often Cognitive Architecture


Other fields of interest

• Psycholinguistics

• Social cognition

• Cultural Psychology

• Communication theory

• Embodied communication,paralinguistics

• Anthropology

6


Example of metacognitive gap

• The “computer metaphor” of mind

• Intuition: Single mental processor reads all

senses and performs a variety of tasks.

• Cognitive Science: Processing modules

operate in parallel.

– Restricted flow of information and control.

– Processing characteristics are counterintuitive


“Horizontal” Modularity

• Model Human Processor (MHP)

• Serial stages of processing

• Information flow is Bottom-up

– Cognitive impenetrability

– “Seeing is believing”

Action

Cognition

Perception


Horizontal modularity restrictions

Cognitive impenetrability (Pylyshyn,1984) refers to the inability of observersto use semantic information (such aswhat the person believes or intends to do)to influence the operation of the inputstage.


Vertical modularity (Fodor)

Phoneme perception

Voice recognition

Auditory localization

Cognitive processing

7


Vertical modularity restrictions

Information encapsulation (Fodor, 1983)refers to structural barriers within the

cognitive architecture that prevents internaldata stores from being shared between

modules in the same stage of processing.


Events are processed by many channels

• Intuitions about thinking

– Fails at low levels

• Cognitive Architecture

– Multiple brain areas

– Interconnected

– Informationally encapsulated

– Multimodal inputs parsed from scene

Cognition

DorsalSystem

Bimodalspeech

Ventralsystem

…

Sensory world


Modularity of reading

• Name the colour of the text

• Respond as quickly as possible

• Measure response time

• 2 trials


DogDog

CatCat

FishFish

BirdBird

CowCow

HorseHorse

PigPig

CowCow

FishFish

8


GreenGreenRedRed

OrangeOrangeRedRedBlueBlueBlueBlue

OrangeOrangeGreenGreenRedRed


Cognitive Impenetrability & Modularity

• Stroop effect– Reading is data-driven module– Competition for response

• Other Modularity phenomena– Modularity of perception for action– Modularity of visual/auditory integration– Modularity of eye movement control– Modularity for Models of Minds


Advantages of modular processing

• Download task to module, reducecognitive bottlenecks

• Fast, effortless information processing

• Near-optimal information integrationbetween cues and sensory modalities– Fuzzy logic cue integration

– Bayesian categorization


"Civilization advances by extending thenumber of important operations whichwe can perform without thinkingabout them."

Alfred North Whitehead

What are the disadvantages?

9


Disadvantage: Processing rigidity

• Stress within a module is not accessible• Complex stimuli may be processed

differently in different modules• Tasks may access different modules with

different performance characteristics• Virtual environments may introduce

discrepancies that impact different modules(and tasks) differently


Stress within a module may beundetected

Poor cognitive access tolow-level processes

Action

Cognition

Perception


Example: VDT stress syndrome

• Users complain of headache, visionproblems etc.

• Reports anecdotal, but readingimpairment is observed

• Also pupillary tremor and regressivesaccades


Reflective HCI Practice (after Schön)




Evaluation, Mapping

Implement

Interaction Design



10


Reflective HCI Practice: Foraging






Theory of space constancy in activevision

• Cognition needs the illusion of a stable world

• Eye movements should create confusingimage shifts

• Maintaining space constancy requires– Efference copy

– Passive blur

– Lateral masking

– Saccadic suppression


Reflective HCI Practice: Hypothesizing




Evaluation, Mapping




VDT fatigue study

• Hypothesis “Sampling” raster duringsaccades reduces intra-saccadic blur, andmay overload saccadic suppression

• Space constancy perspective allowed us to:– Isolate the important factors in a complex

situation

– Find a more sensitive task and measure

• Study examines detection of movementduring saccade (w Bridgeman & Macnik)

11


VDT fatigue study recommendations

• Suppression thresholds elevated forflickering stimuli

• Effect is reduced >250 Hz

• Problems will be greatest– large saccades

– high-contrast display

• Work arounds include blanking displayduring saccades


Reflective HCI Practice: Testing




Evaluation, Mapping

Implement

Interaction Design




Disadvantage: Each module mustsolve feature assignment problem.

• Modules can’t accept information from othermodules: Information encapsulation

• Different modules should have access to adifferent set of matching cues.

• Illusory conjunctions can occur inmultimodal environments:– Phoneme perception: The McGurk effect

– Auditory localization: The ventriloquist effect


Study: Impact of display errors onmultimodal perception

• Immersive environments typicallyhave display errors– Location of events is not precise– Timing is not precise– Graphics can be low-fidelity

• As immersive environments addsound and touch, what will be theimpact of these errors?

12


Reflective HCI Practice:Foraging






Recalibration by pairing (Epstein, 1975)

• Individual senses adapt to display• Sensory modalities calibrate each other:

haptics, vision, sound– Observed actions calibrate visual space (space

constancy)– Vision calibrates hearing for the location of a

multimodal event– Sound calibrates vision for the time of a

multimodal event

• Result is an after-effect: remapping ofauditory (visual, haptic) space


Reflective HCI Practice: Hypothesizing




Evaluation, Mapping




Impact of information encapsulation

• Multimodal environment with errors intiming and location

• The same event might give rise to a singlemultimodal construct in one task, and twounimodal events for another.– Vary location of visual and auditory phonemes

in a simple teleconferencing-style video display

– Vary information carried by using syntheticspeech stimuli (5 levels).

13


Ventriloquism meets the McGurkeffect.

• Vary location of visual and auditory phonemesin a simple teleconferencing-style video display

• Vary information carried by using syntheticspeech stimuli (5 levels).

• Subjects report sound location and syllableheard,

• Analyses included testing a variety ofmathematical models of information integrationby fitting free parameters with STEPIT.


Use of mathematical modeling toolsallow us to address

• Sensory input from a number of channelssimultaneously

• How stimuli from multiple channels arematched and partitioned into mentalrepresentations

• How information from multiple senses isintegrated to give rise to trans-modalmental events


Reflective HCI Practice: Testing




Evaluation, Mapping

Implement

Interaction Design




Results:

• Visual capture of auditory source location, resultingin a shifting of unimodal auditory locationestimation (ventriloquism after-effect).

• No effect of location difference on phonemeperception as measured by statistical or modelingtests.

• No correlation between errors in the two tasks (i.e.subjects could not selectively attend to the auditoryphoneme on trials when visual capture failed).

• Overall, modularity of phoneme perception issupported.

14


Disadvantage: Changes in task interactwith modules to change performance

• 2 visual systems—“ventral stream” forrecognition and “dorsal stream” for action.

• Where vs how

• Different impact of illusions• Lesion data


Functional Neuroanatomy ofperception for action.

2 visual systems—“ventral stream” for cognition and“dorsal stream” for motor performance.


2 visual systems lesion evidence

identification, shaperecognition, objectorientation

object manipulation(orientation matching,grip scaling)

Posteriorparietal (RV)

object manipulation(orientation matching,grip scaling)

identification, shaperecognition, objectorientation

Ventrolateraloccipital (DF)

pointingdetection andidentification

V1 (blindsight)

spared abilitiesperformancedeficits

lesion


2 visual system illusions

pointinginduced motion,location report

Moving or off-centre frame

pointingdetection ofdisplacement,location report

displacementduringsaccade

grip scalingsize reportTichner circles

spared abilitiesdeficitsstimuli

15


Research with videoconferencingand abstract displays

• Targeting sound: cognitive better thanmotor– Subs aware of visual and auditory

locations, but point to visual

• Targeting vision with context: Lessfeedback is better– Pointing with no visual feedback better

– Lagged cursor better than unlagged


Interpreting pointing studies

• Pointing studies counterintuitive, but

predicted by response characteristics

of neurons in dorsal/ventral to visual

and auditory stimuli

• More in “Graphics in the Large” panel,

Thursday 3:30


2 visual system illusions

apparent location ofsound

pointingSound withdisplacedvisualdistractor

pointinginduced motion,location report

Moving or off-centre frame

pointingdetection ofdisplacement,location report

displacementduringsaccade

grip scalingsize reportTichner circles

spared abilitiesdeficitsstimuli


Modularity Recap

• Displays can interact with active perceptionto cause hidden stress: VDT study

• Displays can impact matching cues formultimodal cue integration– Support or frustrate recalibration by pairing– Lead to discrepancies in number and

composition if stimuli in different modules• Displays can cause disagreement between

motor performance and cognitive measures• Sometimes removing information helps

16


Extending to complex worlds

• Previous studies in simple worlds, with afew visual and auditory events

• Multimodal environments are complex– Virtual worlds

– Augmented reality

– Ubiquitous computing

• How are multiple multimodal events dealtwith in the cognitive architecture?


Indexical cognition (Pylyshyn)


“FINSTs... make thoughts true”

• Perception– “Hotlink” tokens– Drawn to salient events– Object-centred, “sticky”– Visual routines– Finite number ~ 4

• Cognition– Maintain object history– Implicit memory of object associations– Sparse cognitive representation– Just-in-time delivery of information– Atom of intentionality


Mental representations of complex worlds

• Cognitive architecture perspective requiresthat links be established between lowerlevel perceptual qualities and cognitivesymbols—i.e. a pointer, called a FINST.

• FINSTing allows us to interact withperceptual objects and events without theneed for mental images per se.

• Symbolic representation + pointers makesdifferent predictions than intuitive picture-in-the-head

17


Indexical cognition (Pylyshyn)


Naïve view of FINSTs in Cognitive Arch

Phoneme perception

Voice recognition



Action (motor space)

FINSTs


Another view of FINSTs

Phoneme perception

Voice recognition



Action (motor space)

FINSTs


More about FINSTs

• FINSTs Link mind & perceptual world– Visual routines: (collinear, inside,

subitizing)– History of an object– Object-centred, “sticky”– Drawn to salient changes-- onsets,

luminance increments, oddballs– Finite number ~ 4-7– FINSTs + ANCHORs for motor behaviour

18


More about ANCHORs

• ANCHORs link mind & action– Remembered locations for eye

movements

– Direct interaction with items off the retina

– Fast, robust motor performance by actionroutines

– Affordances for action


Multiple object tracking demo


Another trial (Scholl)


Another trial (Scholl)

19


Application


Mental representations of complexenvironments

• Cognitive architecture perspective requires thatlinks be established between lower level perceptualqualities and cognitive symbols—i.e. a pointer,called a FINST.

• FINSTing allows us to interact with perceptualobjects and events without the need for mentalimages per se.

• Symbolic representation + pointers makes differentpredictions than intuitive picture-in-the-head

• Coping with spatial transformations in complexdata spaces


Module advantages

• High-realism interface designs improve

performance by “downloading” information

processing operations to input modules.

• Interaction of display characteristics with

capabilities and characteristics of the

functional architecture will determine

performance.


Module disadvantages

• Coordination– Distortions in location, timing, and category-relevant

information may lead to the formation of conflictingrepresentations in different modules.

• Processing inflexibility– Errors and conflicts within a module can create errors and

increase cognitive load. (CRT flicker example)

• Information hidingCognitive impenetrability of modules makes it difficult for

operators to determine the reasons for their poorperformance.

20


Inseparability of Mind & World

• Embodied cognition-- mind/body

• Situated cognition-- mind/world

• Distributed cognition-- mind/mind

• Ecological theories (Vygotski, Luria,Bateson, Gibson) can be linked tosensory phenomena and informinteraction design


Key Points for HCI Practice

• The Metacognitive Gap: The need for agrounded Cognitive Science approach

• Reflective design practice methods– Integrating CogSci with interaction design– Iterative design cycle

• Examples of extended HCI– Cognitive architecture: Multimodal displays and

how they are understood– Situated cognition: embodied interaction with

complex displays


What to expect in the next talk

• More about perceptual and attentivebottlenecks

• Change blindness

• Three flavours of (in)attention

• Implications for visual displays

• Coercive graphics


Environment/Inhabitant/Representation

• Information visualization

• Perceptual/deictic/situated Cognition

• Cognitive Architecture, perception, attention

• Display as extension of mental model

21


Thirteen Ways of Looking at a BlackbirdWallace Stevens

I

Among twenty snowy mountains, The only moving thing Was the eye of a blackbird.

II I was of three minds, Like a tree In which there are three blackbirds.

III The blackbird whirled in the autumn

winds. It was a small part of the pantomime.

IV A man and a woman Are one. A man and a woman and a blackbird Are one.

V I do not know which to prefer, The beauty of inflections Or the beauty of innuendoes, The blackbird whistling Or just after.

VI Icicles filled the long window With barbaric glass. The shadow of the blackbird Crossed it, to and for. The mood Traced in the shadow An indecipherable cause.

VII O thin men of Haddam, Why do you imagine golden birds? Do you not see how the blackbird Walks around the feet Of the women about you?

VIII I know noble accents And lucid, inescapable rhythms; But I know, too, That the blackbird is involved In what I know.

IX When the blackbird flew out of sight, It marked the edge Of one of many circles.

X At the sight of blackbirds Flying in a green light, Even the bawds of euphony Would cry out sharply.

XI

He rode over Connecticut

In a glass coach.

Once, a fear pierced him,

In that he mistook

The shadow of his equipage

For blackbirds.

XII

The river is moving.

The blackbird must be flying.

XIII

It was evening all afternoon.

It was snowing

And it was going to snow.

The blackbird sat

In the cedar-limbs.

modularity and multimodal interaction brian fisher …fisher/siggraph2002coursenotesbf.pdf5 fels:...

Documents