talk @ macquarie uni - workshop on distributed cognition

8/2/2019 Talk @ Macquarie Uni - Workshop on Distributed Cognition

1/32

Transforming the Enculturated Mind:Sensory Substitution and

Complementarity

Mirko [email protected]

Cognitive Science ARC Centre of Excellence in Cognition

and its Disoders
mailto:[email protected]:[email protected]


2/32

Complementarity defences of

EMT argue that many of thekinds of cognition humansexcel at can only be

accomplished by brainsworking together with a bodythat directly manipulates and

acts on the world

Complementarity in a Nutshell

I take SSDs as my empiricalcase study to explore and

illustrate the ramifications ofComplementarity


3/32

In the first partof this talk, Iquickly lookat the issue

of whetherthere is trulysubstitution

(either visual

or tactile) ornot

Today s Talk


4/32

I argue that there isno real substitutionbut rather cognitive

and perceptualsupplementation

I finish up by relatingmy conclusions to

the idea of

Complementarity


5/32

So, the Idea

Sensory substitution basicallymeans to transform the

characteristics of one sensorymodality into stimuli of another

sensory modality

The principles of sensorysubstitution have been

formulated by Bach-y-Rita, who

conducted experiments withthe potential of the skin as amedium for transmitting

pictorial material


6/32

Visual-to-Tactile Substitution Devices


7/32

Visual-to-Auditory Substitution Systems


8/32

What type ofperceptual

experience thepracticed user of

a sensorysubstitution

device can be

said to undergo?

SSDs raise many interesting philosophical questions


9/32

Keeley believes that possessing an authenticsensory modality involves the acquisition of a

genuine, wired-up (dedicated) sense organ

Keeleys Dedication

This organ has to be

phylogenetically developed tofacilitate survival with respect to an

identifiable class of phenomena


10/32

Dedication and SSD perception

It is true that through the coupling with the device the

impaired user receives visual information about the world, butshe discriminates such stimuli behaviourally, via atactile/auditory capacity!

Providing blind individuals with an SSD doesnt suffice to endow

them with a sensory modality they did not have before.


11/32

For Keeley, the eyes are necessary for seeing andnothing can see that doesnt have a sense that hasevolved for detecting properties via light. The coupling

with the SSD only allows the agent to jerry-rig asensory system dedicated to the reception of

mechanical distortion (his skin) into one capable ofproviding him with generally reliable information about

the electromagnetic spectrum . [Keeley (2002),p.20].

SSD provides the impaired users with informationbut only via a dedicated (tactile/auditory) channel

that has already evolved to detect properties inthe world.


12/32

Spatial Encoding & Cognitive Inference

Prinz and Block also acceptthat the SSD perceiver hasexperiences with spatial

significance but they denythat this spatial significance

is visual in character

They concede that SSDs encode spatial contents but argue

that this only enables the visually impaired to use somefeatures of the proximal stimulus to make cognitive inferenceson the basis of dedicated neural pathways


13/32

Is it rather Vision?

Hurley and No

have argued thatafter substantial

training andadaptation the

phenomenologyof the perceptionobtained throughthe coupling withan SSD switches

fromtactile/auditory

to visual


14/32

Depends on the amount ofsensorimotor contingencies thatthis acquired perception shares

with natural vision

The more the user masters thedevice, the more invariants heracquired perception shares withvision. The more invariants the

acquired perception shares withnatural vision the more it

resembles it.

The nature of SSD perception


15/32

Is it TRULYsubstitution?

I argue it isnot !

It is a supplementation


16/32

Neither Touch nor Vision

Auvray et al. (2007) the conveyed qualitative experience is

not automatically associated to either audition/touch or vision butrather reported to occur as something entirely new, whose nature

was essentially task-dependent

tactile sensation persists over time AFTER TRAINING - veridicalrepresentations of things out there in a

three-dimensional space

This new type of experience doesnt entirelyqualify as tactile nor exclusively as visual, but

possesses both components


17/32

Quasi Vision?

Through the coupling with SSDsthe visually impaired gets a mode

of access to the world thatdepends vertically on pre-existing

modes of perception whilstnevertheless counting assomething entirely new

This new mode of accessemerges from users pre -

existing sensory modalities,and its novelty is determinedby the fact that it no longer

aligns with them


18/32

Something new: but what exactly?

If the phenomenology of SSD perceptiondoesnt stay in one modality but exploits

the pre-existing senses to give ussomething new, couldnt we just

speculate that SSD perception, in givingthe visually impaired something new,blends vision with hearing or touch?

So SSD perception, stands at anew level above the pre-existing

perceptual modalities and itsvarious sensory divisions


19/32

Artificial Synaesthesia?

Synaesthesia iscondition in whichstimulation of one

sensory or

cognitive pathwayleads to automaticand unintentionaloccurrences in a

second sensory orcognitive pathway.


20/32

Synaesthesia isoften describedas a merging ofthe senses, across-modal

union of different

sensorymodalities!


21/32

The Neuroscience Behind Synaesthesia

To explain the neurocognitive mechanisms that characterisethis phenomenon Cohen Kadosh, Walsh, and Henik have

suggested that synaesthesia is due to disinhibition orunmasking of signals between or within brain areas


22/32

The Neuroscience behind SSDs (1/2)

Amedi et al. (2007) have shownthat the LOtv is involved inshape extraction/recognition

from visual-to-auditory soundscapes

Interestingly, this is reported totrigger cross-modal experiences

where auditory/tactilestimulations are combined with

visual elaboration


23/32

Kupers andcolleagues (2011)have recently

favoured an accountof cross-modal

plasticity in SSDusers that involves

disinhibition ofexisting pathways

over a view thatprescribes corticalreorganisation

The Neuroscience behind SSDs (2/2)


24/32

If cross-modalplasticity in SSD users

is explained in termsof disinhibition and thisform of disinhibition or

unmasking alsocharacterises thephenomenon of

synaesthesia, couldntwe propose that aform of artificially

induced synaesthesiacan occur in SSD

perception?


25/32

A Report:

Monochrome artificially induced synaesthesia in certain frequencies of sound...The thing I experience is not in color, is in my mind's eye, and can be very distracting.The shapes are consistent and can be reproduced by the same sound. It is almost as if you had a computer with two monitors running simultaneously different pictures, one was a very grey blurred version of the real world, and the other was a pure grey background with a big semi-circular light grey arc on it, and sometimes you

switched your attention between both. [Ward & Meijer(2010),p.497-498].


26/32

Stability over time: a hallmark of Synaestetic Experience !

Interestingly, subjectsclaim to see sounds

even when not wearingthe device.

Their brain has internalizedthe vOICe rules for mappingbetween hearing and vision

and these rules aredeployed, by virtue of

mental imagery, both whenthe device is worn and

when it is not


27/32

A Long Journey Back Home .

This novel sensorimotor coupling

triggers an experience that isquite original

SSDs systematically transform thesensory experience of the impaired,

by providing a novel perceptualmodality that compensates for loss

or impaired sensory channel

Mind Enhancing Tools, at leastfor the visually impaired [Clark

(2003)]


28/32

Auvray &Myin (2009)have argued that,such devices should not be understoodas merely external stand-ins for alreadyexisting purely internal processes ... butrather taken to transform cognition andperception in a qualitative way [ Auvray

& Myin (2009), p.1051]; in a way thatwould otherwise be precluded to the

impaired non-SSD user

Cognitive Reengineering


29/32

SSDs are therefore an example of cognitive andperceptual transformation, achieved via acquisition

of embodied expertise

Through learning in fact, they get factored andintegrated into the impaired users perceptual

processing and become a different butcomplementary part of the machinery that realises

her cognitive capacity


30/32

Thus SSDs, via plasticity,provide the visually impairedwith the means for expanding

perception towards newhorizons

SSD perception isnt a meresubstitution but rather an

addition, a supplementation orbetter a complement

The result of this complement isa biotechnological synthesisthat entails the creation of a

new space of coupling betweena human being and the world


31/32


32/32

Special Thanks to:

John Sutton Richard Menary Julian Kiverstein Andy Clark Greg Downey Malika Auvray Jack Loomis Peter Meijer

talk @ macquarie uni - workshop on distributed cognition

Documents