a framework for perceptive, attentive, cognitive, and social-interaction enhancement from...

A Framework

for

Perceptive, Attentive, Cognitive,

and

Social-Interaction Enhancement

From User-Centred

to

Ego-Centred Design

Morris Goldberg

PACSIE

Sreekar KrishnaProf. Panchanathan

Perc

ep

tion C

og

nitio

n

Attention

Social Interaction

•Perception:• Hear a baby cry.

•Attention:•Listen so as to hear one’s baby cry

•Cognition:•Identify the reason for crying

•Social-Interaction:•Rock the baby to stop the baby from crying

Friday, April 21, 2023 PACSIE: Ego-Centred Design 3

Perception, Attention, Cognition, Social-Interaction

•Movement:•Turn head to better hear the baby cry

•Move towards the baby, inspect baby

•Pickup baby, rock the baby

•Autonomic:• Posture: as baby may cry

•Heart-rate: if baby cries


Behaviour Responses:Movement and Autonomic

•Applications• Cerebral: injury, disease, and aging

• Autism, prosopagnosia

• « Extra eyes »

•Training, drugs, invasive• Practice makes perfect,…

•Assistive device• Non-invasive, customized, adaptive, lifelong


Enhancement


Motivation: The Carpenter Who Lost His Tools

Visual agnosia for toolssees tool but lost knowledge of tool

name, function, memory

Haptic tool recognition feels tool, knowledge recovered

Infancy: body(hands) calibrates vision

Maturity:

vision: dominates

body:mostly below conscious awareness

•CO poisoning heater•« Blind » to many objects•Recognized her mother through voice•Recognized colours, texture•Name some surafces

–Red plastic, shiny metal•Spatial acuity intact

–Gratings, but not directions•Basic problem recognizing shapes

•MA Goodale, AD Milner, Oxford Univ. Press

–Sight Unseen: An Exploration of Conscious and Unconscious Vision, 2004Friday, April 21, 2023 PACSIE: Ego-Centred Design 7

Patient DF: Visual form agnosia


Patient DF: Tests

• Object recognition (haptic)

• Copying objects

• Yellow blind to orientation

• Grasped any orientation


Visual processing streams: what, where, how

•Visual agnositics• rare, encumbered, difficult, non-stable

•The blind• visualize objects by touch, stable, intelligent

•Reader• NSF, focus groups


Surrogates: The visually impaired

•Vibro-tactile glove •Sensory substitution: distal visual to proximal haptic

•Haptic rendering not possible

•Observe haptic manipulation

•Uncover distinguishing haptic features/descriptors

•Deliver features by haptic cueing

•Supervised user learning: haptic cues

•Object visualization: user imagination


Haptic object visualisation

Kahol K., Panchanathan S.,“Neurocognitively inspired haptic user interfaces”,

Springer J. of MM Tools and App, v 3, pp 15-38 2007.

Kahol K, Smith M., “Neuro-cognitively Designed Dynamic Simulations For Laparoscopic Surgical Skills”. MODSIM. Virginia Beach, 2007. pp. 34-39.

•Design methodology •Participatory design

focus groups -scenario building

•Prototypic User •Average user

•Learning •Designer discovery

usage patterns, blog, competition..

•User discovery

short-cuts, hidden functionality


User-Centred Design

•Assistive device• white cane.. extension of self…peripersonal space

• left-neglect…stroke…line bisection..finger, laser-pointer, ,stick

•Haptic exploration • haptic glove 6 blind people pattern varies

• depends on degree and onset of blindness, other factors

•Ego-centred design• understanding of the individual

perceptive, attentional, cognitive, social (PACS)


Framework

•1st Principle: Sensory-Motor Coupling

•2nd Principle: Polymodal Reinforcement•Edelman, GM, « Neural Darwinisim: The Theory of Neuronal Group Selection », 1989.

•3rd Principle: Graft Onto Existing Scaffolds•Thelen and Smith, “A Dynamic Systems Approach to the Development of Cognition and Action”, 1996.

•4th Principle: Self Evaluation


Ego-Centred Design: Principles

Active sensing sensory adaptation to constant signal

adapt to pressure on body, smell, even vision

shift body, sniff, eye saccade, ….

« What » is being sensed? behavior is visible

deduce « what »

haptic manipulation- descriptors

Haptic rendering motor-sensory coupling

individual moves, assistive devices provides stimuli


1st Principle: Sensory-Motor Coupling

Modes vision: colour, shape, orientation, motion….

somatosensory: light and heavy pressure, pain, vibration…

auditory, proprioceptive, olfactory:…….

Existence-pathological cases Agnosia: loss of mode/function

Synaesthesia: extra function-e.g see sound, hear color


2nd Principle: Polymodal Reinforcement

Examples texture: somatosensory, visual, and auditory

taste: smell and somatosensory(tongue)

Associative memory Bartlett: memory is a re-creative process

Move hands, visualize, hear , feel



Exploitation Wagner: musical identities

rehabilitation: colour code objects, audio prompts

Haptic glove audio and haptic cues for textures



Child development reaching: months of effort, muscles, vision proprioceptive

orienting to people: instintive attraction to face-like patterns

Continuumscaffolding: basis for new functionalities

development nevers stops

Blind and body languageauditory cues: nervous movement, breathing, voice inflection

assistive device: correlate/integrate


3rd Principle: Graft Onto Existing Scaffolds

Gait variesload, terrain, physical condition

trial and error: self evaluation (pain, comfort)

Continuous self-evaluation

Innate Behavioral Responses attraction to eye-like objects

move eyes- get reward (smile,hug)-

emotional-based value system


4th Principle: Self Evaluation

Everyday Social InteractionsWhat do we know about them?


Social Interactions: Complex Visuo-Motor Interplay

Social Interactions

SensorySensory

PerceptualPerceptual

CognitiveCognitive

MotorMotor

Handshake

Eye Contact

Move towards each other(Keep in mind Proxemics)

Intent to interact

Shake hands

Conversation distance



Psychological Basis of Social Interactions• Emotions

• Interpersonal Attributes• Dominance• Trust• Like/Dislike

• Personality

• Communication• Turn Taking• Feedback• Attention• Iconic Cues, like Victory or Stop hand gestures.• Intrinsic Cues, like moving close to someone

Beh

avio

ral A

spec

tsB

ehav

iora

l Asp

ects

Com

mun

icat

ion

Asp

ects

Com

mun

icat

ion

Asp

ects

M. Argyle, The Psychology of Interpersonal Behaviour, Penguin UK, 1999.


Psychological Aspects – Dyadic Interactions

FeelFeel SendSend ReceiveReceive FeelFeel

M.G. Almore, “Dyadic Communication,” The American Journal of Nursing, vol. 79, Jun. 1979, pp. 1076-1078.

A. Teeters, R.E. Kaliouby, and R. Picard, “Self-Cam: feedback from what would be your social partner,” ACM SIGGRAPH 2006 Research posters, Boston, Massachusetts: ACM, 2006, p. 138.

ExpressionsBody MannerismsCommunicative Gestures


Insula and the Mirror Neurons

Mirror NeuronsMirror

Neurons InsulaInsula Limbic SystemLimbic System

Simulate Facial

Expression

Feel the Emotion

L. Carr, M. Iacoboni, M. Dubeau, J.C. Mazziotta, and G.L. Lenzi, “Neural mechanisms of empathy in humans: A relay from neural systems for imitation to limbic areas,” Proceedings of the National Academy of Sciences of the United States of America, vol. 100, Apr. 2003, pp. 5497-5502.

Derived out of fMRI studies on human viewing facial images Enhanced activity in the Mirror Neuron Area, Insula and Limbic System observed Findings are just anatomical patterns Exact Pathway is still not discovered

Everyday Social Interactions – The PACSIE Model

Bringing it all together


Friday, April 21, 2023 27PACSIE: Ego-Centred Design

Principle 1: Sensory-Motor Coupling

• Social Interactions are integration of complex sensory and motor actuations and responses.

Enactor (Encoding) Recipient (Decoding)

Visual EngagementVisual Engagement

Coordinated Motor Interaction – Eye Gaze,

Head Nod

Coordinated Motor Interaction – Eye Gaze,

Head Nod

Communication – Asynchronous Exchange

Communication – Asynchronous Exchange

Complex Social Engagement

Complex Social Engagement

PerceptionPerception

AttentionAttention

CognitionCognition

Social InteractionSocial Interaction

Ver

bal

Non

-ver

bal

Enactor (Encoding) Recipient (Decoding)

Body

Voice

Speech

65%

35%

27%19%

18%35%

Face Visual

Audio


Principle 2: Polymodal Reinforcement

• Social Interactions are integration of multichannel information (from the enactor and the recipient’s perspective).

N. Ambady and R. Rosenthal, “Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis.,” Psychological Bulletin. Vol. 111(2), vol. 111, Mar. 1992, pp. 256-274.


Principle 3: Use existing skills (i.e. Scaffolding)

• Social Interactions capabilities are very individualistic based on when the person becomes blind.

Congenitally Blind

Congenitally Blind

Adventitiously Blind

Adventitiously Blind

BirthBirth Developmental YearsDevelopmental Years Professional YearsProfessional Years Geriatric YearsGeriatric Years

Social Skills

Social assistance need varies depending on when the person loses his/her sight.


Principle 4: Recalibration by Self Evaluation

• Social Mirroring is the process by which effective social interactions are learned and propagated.

Hello !!!Welcome to

CUbiC

Hello !!!Welcome to

CUbiC

ActionAction

Social MirrorSocial Mirror

PerceptionPerception

Mirror NeuronsMirror

Neurons

Self Evaluate

Self Evaluate

M. Iacoboni, Mirroring People: The Science of Empathy and How We Connect with Others, Picador, 2009.

Social Solutions Through PACSIEBody Rocking Condition



Role of vision

Learning through

observation

Learning through

observation

Feedback on social

mannerisms

Feedback on social

mannerisms

Visual Impairment affects both aspects of social interactions


Stereotypic Mannerisms

Hand - 60.3%

Head- 13.1%

Trunk - 10%

Body - 6.9%

Leg - 6.4%

Face - 3.4%

Though body rocking is not the

most occurring mannerism, it is the

most noticed in a social scene

V.J. Eichel, “Mannerisms of the blind: A review of the literature,” Journal of Visual Impairment & Blindness, vol. 72, 1978, pp. 125-130.

Body Rocking in Adults

Rocking is a natural development behavior of the cerebellum’s balance system

Body rocking in adults is due to the lack of social feedback



Solutions

InterventionIntervention Self MonitoringSelf Monitoring

Immediate feedbackDoes not address the problemReduces the symptomChance of recurrence is high

Replace with Cognitive TaskProvides for symptom

substitutionChance of recurrence is low

Focus on Controlling the problem and not curing it !!!

M. Celeste, “Social Skills Intervention for a Child Who Is Blind,” Journal of Visual Impairment & Blindness, vol. 101, 2007, pp. 521-533.

R.L. Loftin, S.L. Odom, and J.F. Lantz, “Social Interaction and Repetitive Motor Behaviors.,” Journal of Autism & Developmental Disorders, vol. 38, Jul. 2008, pp. 1124-1135.


Social Solution

PACSIEPACSIE

Body Rocking

Body Rocking

Social Solution

Social Solution

Social Feedback

Social Feedback

Self Evaluate

Self EvaluateControlControl


Social Interaction Assistant: Work-inProgress


Person, Gaze, Expression, Identity Detector

•1st Principle: Sensory-Motor Coupling•Head movement controls sensing(far-near,left-right)

•2nd Principle: Polymodal Reinforcement•vibration and sound

•3rd Principle: Graft Onto Existing Scaffolds•mood recognition, voice recognition, footstep recognition..

•4th Principle: Self Evaluation•user evaluates/calibrates the signals from the SIA

•eventually SIA will also self-evaluate


Ego-Centred Design: Principles

a framework for perceptive, attentive, cognitive, and social-interaction enhancement from...

Documents

egocentred designpacsie

egocentred designmovement

baby cries

babypickup baby

baby crymove

haptic user interfaces

ones baby crycognition

user imagination