SENSORY SUBSTITUTIONDr. Tom Froese

Structure of partial exam• Short written essay• 25% of total course grade• Scientific formatting and referencing• Theme: why new approaches to cognitive science?

• Correct formatting: 50% of this grade• Academic content: 50% of this grade• Plagiarizing: 0%!

• To reproduce or otherwise use (the words, ideas, or other work of another) as one's own or without attribution.

• Deadline: March 17 (3 weeks from now)• Send PDF to t.froese@gmail.com before March 17!

Sensory substitution• Sensory substitution means to transform the

characteristics of one sensory modality into stimuli of another sensory modality.

• A sensory substitution system consists of three parts: a sensor, a coupling system, and a stimulator.

• The sensor records stimuli and gives them to a coupling system which interprets these signals and transmits them to a stimulator.

• Wikipedia “Sensory substitution”

Sensory substitution• It is hoped that sensory substitution systems can help

handicapped people by restoring their ability to perceive a certain defective sensory modality by using sensory information from a functioning sensory modality.

• In case the sensor obtains signals of a kind not originally available to the bearer it is a case of sensory augmentation.

• Sensory substitution concerns human perception and the plasticity of the human brain; and therefore, allows us to study these aspects of neuroscience more through neuroimaging.

• Wikipedia “Sensory substitution”

State-of-the-art • The VEST – “Versatile Extra-Sensory Transducer”• Currently being developed by the Eagleman Lab• Sound-to-touch substitution

Nov

ich

an

d E

agle

man

, un

der

revi

ew

The “Vest”

Novich and Eagleman, under review

The “Vest”

Novich and Eagleman, under review

Many possibilities, many unexplored• Tactile-visual substitution

• TVSS

• Tactile-auditory substitution• The VEST

• Tactile-distance substitution• The Enactive Torch

• Tactile-vestibular substitution• Tactile-magnetism substitution

• feelSpace

• Tactile-tactile substitution• To restore peripheral sensation• Tactile feedback for prosthetic limbs

• Auditory-visual substitution• The vOICe

• …

TVSS• “TVSS” – “Tactile Vision Substitution System”• Bach-y-Rita et al. (1969). “Vision substitution by tactile

image projection” Nature• Video camera -> image processing -> vibro-tactile array

TVSS

TVSS

Bach-y-Rita et al. (1969)

Phenomenology of TVSS (~10hrs)• “As the blind subjects become more familiar with objects,

they learn to recognize them from minimal or partial cues.”

• “Our subjects spontaneously report the external localization of stimuli, in that sensory information seems to come from in front of the camera, rather than from the vibrotactors on their back.”

• “Thus, after sufficient experience, the use of the vision substitution system seems to become an extension of the sensory apparatus.”• Bach-y-Rita et al. (1969, p. 964)

TVSS

Phenomenology of TVSS (3 weeks)• “Inasmuch as I am congenitally blind, I needed to gain

access to visual space or, if not visual space itself, then to something resembling it sufficiently…”

• “The Tactile Vision Substitution System provided me with this opportunity.” …

• “it is impossible to pass over in silence the dramatic impact of gaining access to a whole new world of sensory experience.” …

• “the experienced quality of the sensations was nothing like that perceived by touch. Rather than coining a new word, or using ‘touch’ or ‘feel’, I shall use the word ‘see’ to describe what I experienced.” (Guarniero 1974, p. 101)

Phenomenology of TVSS (3 weeks)• “Only when I first used the System did the sensations seem as

if they were on my back. Later on, the sensations appeared to me to be in a two-dimensional space which did not seem to be ‘out there’.” (p. 101)

• “The high point of the experiment occurred when I ‘saw’ myself for the first time.” (p. 104)

• “Very soon after I had learned how to scan, the sensations no longer felt as if they were located on my back, and I became less and less aware that vibrating pins were making contact with my skin.” (p. 104)

• “By this time objects had come to have a top and bottom; a right side and a left; but no depth – they existed in an ordered two-dimensional space, the precise location of which has not yet been determined.” (p. 104)

TCNL, University of Wisconsin-Madison

Tactile Vestibular Substitution Tactile Vision Substitution

BrainPort (Bach-y-Rita et al.)

Electrical stimulation to the mouth!TDU (Tongue Display Unit)

BrainPort

Left: camera, right: TDU

The vOICe (Oh! I See!)

Meijer (1992)

The vOICe

Learning to perceive with the “vOIce”

Auvray et al. (2007)

“analysis from a subjective perspective yielded insights into participants' qualitative experience and into the strategies they used to master the device, and thus to pass from a kind of deductive reasoning to a form of immediate apprehension of what is being perceived.”

Phenomenology of the vOICe• ‘‘I cannot tell fine little tiny details. Rather my vision is based upon black

and white and all the little gradients in between. The best way I try to describe this to people is: take a large black sheet of paper; now take a magical piece of white chalk and sketch me here on this stage in a line drawing; now make me three dimensional and you’ve just about represented how my sight looks.” (p. 495)

• ‘‘I’ve developed it where one day I was washing dishes and without thinking I grabbed the towel, washed my hands, and looked down into the sink to make sure that the water had got out and I realised ‘‘Oh! I can see down. I can see depth.” And I stepped back from that sink and looked down again to make sure that I wasn’t fooling myself , and I walked slowly through my house looking into the rooms and it was like incredible. I could see into the room. That flat drawing now has depth to it. I can sense it. I can’t tell you what is across the other end of the room but I can tell you that it is across from me.” (p. 496)

Ward and Meijer (2010)

Acquired synaesthesia for non-vOICe stimuli

• ‘‘Monochrome artificially induced synaesthesia, only in certain frequencies of sound. A small price to pay for very detailed vision, but the consultant’s music next door sets me off as well (Bach Mass in B Minor). . . The thing I experience is not in color, is in my mind’s eye, and can be very distracting. The shapes are consistent and can be reproduced by the same sound. It is not triggered by all sounds but by vOICe-like sounds (the program, not people’s voices). It is almost as if you had a computer with two monitors running simultaneously different pictures, one was a very grey blurred version of the real world, and the other was a pure grey background with a big semicircular light grey arc on it, and sometimes you switched your attention between both. The arc picture was triggered by the sound of a police car going by my office.”

• ‘‘high small flat white flashing disk” (pure tone of 2000 Hz, 500 ms duration).• ‘‘white circle on left circle is rotating. Reminds me of looking down or into a well or cave.”

(pure tone of 150 Hz, 500 ms duration).• ‘‘clear hollow moving squares middle of face movement left to right.” (noise between 250

and 300 Hz, 500 ms duration).Ward and Meijer (2010, p. 498)

Cross-modal influences• For one subject tactile experiences (e.g. of manipulating

objects) also trigger visual experiences. She claims that these experiences have only emerged after using The vOICe.

• “JW: If you were to touch things you wouldn’t get any visual experiences through that?

• PF: Yes, I can! If I pick up a pencil, I feel pencil, I see pencil.

• JW: Even if you touch it? If you touch an object with your hands you have an experience of seeing it?

• PF: Yes! Touch is vision.”

Ward and Meijer (2010, p. 498)

feelSpace

The Magnetic Perception Group

feelSpace: “Beyond sensory substitution”

• “After six weeks of training we evaluated integration of the new input by a battery of tests. The results indicate that the sensory information provided by the belt • (1) is processed and boosts performance, • (2) if inconsistent with other sensory signals leads to variable

performance, • (3) does interact with the vestibular nystagmus and • (4) in half of the experimental subjects leads to qualitative changes

of sensory experience.

• These data support the hypothesis that new sensorimotor contingencies can be learned and integrated into behaviour and affect perceptual experience.”• Nagel et al. (2005, p. R13)

Phenomenology of feelSpace• Some subjects described that the input from the belt reflected

properties of the environment rather than being simply tactile stimulation: • “It was different from mere tactile stimulation because the belt mediated a

spatial feeling.”• “I was intuitively aware of the direction of my home or of my office. For

example, I would wait in line in the cafeteria and spontaneously think: I’m living over there.”

• Unexpectedly, magnetic north had no special status, but spatial perception related always to landmarks. The actual spatial context was felt as being massively enlarged, and spatial relationships could be memorized effortlessly. One subject travelled to foreign cities with the belt and reported: • ‘The orientation in the cities was interesting. After coming back, I could

retrieve the relative orientation of all places, rooms and buildings, even if I did not pay attention while I was actually there.’

Nagel et al. (2005, p. R22)

Phenomenology of feelSpace• access to the information provided by the belt did not

require attention: • ‘During the first two weeks, I had to concentrate on it; afterwards, it

was intuitive. I could even imagine the arrangement of places and rooms where I sometimes stay. Interestingly, when I take off the belt at night I still feel the vibration: When I turn to the other side, the vibration is moving too—this is a fascinating feeling!’

• Interestingly, for one subject the effects during training were noticeable but not overly impressive. Only after he stopped wearing the belt was the full amount of induced changes apparent:• ‘After removing the belt, my living space shrank quickly: the world

appeared smaller and more chaotic because relative positions to places beyond the visual horizon were rather unorderly.’

Nagel et al. (2005, pp. R22 - R23)

Phenomenology of feelSpace• “Remarkably, in no case are subjects reported to perceive a local

magnetic field. Strictly speaking, the changes in perception indicate not a genuinely new modality, but modification of the meta-modality of spatial perception.

• The term ‘metamodality’ is used to reflect the fact that normal spatial perception is fuelled by visual, auditory and somatosensory information.

• The ability to infer spatial information from these pooled ‘primary’ modalities may have thwarted our objective to create a completely new sensory modality.

• Instead, the acquired sensorimotor contingencies lead to a transformation of this already existing meta-modality.”

Nagel et al. (2005, pp. R24)

Space perception via feelSpace• ‘‘It happens more and more often that I know about the

relations of rooms and locations to each other, of which I was not previously aware.’’

• ‘‘In a lot of places, north has become a feature of the place itself.’’

• ‘‘Mental maps have a range. Now that my maps have all been newly realigned, the range of the maps has been much increased. From here I can point home – 300 km – and I can imagine – not only in 2D bird’s eye perspective – how the motorways wind through the landscape.’’

• ‘‘Space has become wider and deeper. Through the presence of objects/landmarks that are not visually apparent, my perception of space extends beyond my visual space. Previously, this was a cognitive construction. Now I can feel it.’’

Kaspar et al. (2014, p. 53)

Belt perception via feelSpace• ‘‘Often I do not perceive the vibration any more. It is rather a direct

feeling of knowledge – not even really a perception. It does not feel like any other sense.’’

• ‘‘For instance, it might be more useful to know that it vibrates 10–15 on the right of the belly button means one stands in a North to West direction.’’

• ‘‘I feel phantom vibrations on my stomach – similar to as if one was wearing a cap all day long and during night one still thinks it would be on one’s head. I also have the feeling to still hear the sound of the belt.’’

• ‘‘I trained walking a path with the belt with closed eyes. With eyes open it naturally worked but the closed eye training definitely also trained my sensibility for the belt signal.’’

Kaspar et al. (2014, p. 54)

Effects on navigation• ‘‘I have the feeling that my spatial sense of orientation

improved since wearing the belt/since training.’’

• ‘‘With the belt it is easier for me to orient myself in a new environment than without the belt.’’

• ‘‘Since I train my orientation it is easier for me to orient myself in a new environment.’’

• “When I take off the belt my spatial sense of orientation decreases.’’

Kaspar et al. (2014, p. 54-55)

Knowing or perceiving?• ‘‘I notice that it is really difficult to describe the changes

that occur. • I was rarely in situations where I am missing words or

where I encounter so clearly my limits of my competence to find the words.

• Often I don’t know whether I should talk about perception or knowledge regarding the belt. It is this problem which makes it so exciting and thrilling.’’• Kaspar et al. (2014, p. 58)

• Maybe the distinction between knowing and perceiving is somewhat artificial? Both are forms of sense-making.

Tactile-distance substitution

“Haptic Radar”

Haptic Radar – extended skin

The Enactive Torch

Froese et al. (2012)

Homework

Please start reading:

Froese, T., McGann, M., Bigge, W., Spiers, A., & Seth, A. K. (2012). The enactive torch: A new tool for the science of perception. IEEE Transactions on Haptics, 5(4), 365-375

References• Auvray, M., Hanneton, S., & O'Regan, J. K. (2007). Learning to perceive with a

visuo-auditory substitution system: Localisation and object recognition with 'The vOICe'. Perception, 36(3), 416-430

• Bach-y-Rita, P., Collins, C. C., Saunders, F. A., White, B., & Scadden, L. (1969). Vision substitution by tactile image projection. Nature, 221, 963-96

• Froese, T., McGann, M., Bigge, W., Spiers, A., & Seth, A. K. (2012). The Enactive Torch: A new tool for the science of perception. IEEE Transactions on Haptics, 5(4), 365-375

• Guarniero, G. (1974). Experience of tactile vision. Perception, 3(1), 101-104• Kaspar, K., König, S., Schwandt, J., & König, P. (2014). The experience of new

sensorimotor contingencies by sensory augmentation. Consciousness and Cognition, 28, 47-63

• Meijer, P. (1992). An experimental system for auditory image representations. IEEE Transactions on Biomedical Engineering, 39(2), 112-121

• Nagel, S. K., Carl, C., Kringe, T., Märtin, R., & König, P. (2005). Beyond sensory substitution - learning the sixth sense. Journal of Neural Engineering, 2, R13-R26

• Ward, J., & Meijer, P. (2010). Visual experiences in the blind induced by an auditory sensory substitution device. Consciousness and Cognition, 19(1), 492-500