visual soundscapes from your augmented reality glasses& augmented reality overview •...

1

CONGres 2012 - Visual soundscapes from your augmented reality glasses

E-mail: [email protected] (Peter Meijer) URL: http://www.seeingwithsound.com

© 2012 Peter B.L. Meijer

CONGres 2012October 17 2012, Amsterdam

Sounds in Science

Peter Meijer, Metamodal BV

www.seeingwithsound.com

visual soundscapes

from your augmented reality glasses

0%


2

you in 2014?you in 2014?

camera

displaydisplay

2%

2




3

?4%


4

visual soundscapes

& augmented reality

Overview

• introduction

• sensory substitution

• visual soundscapes

• brain plasticity

• augmented reality

• conclusions

Overview

• introduction





• conclusions

cameraglasses

6%

3




5

visual soundscapes

& augmented reality

Overview

• introduction





• conclusions

Overview

• introduction





• conclusions

cameraglasses

9%


6

bionic eye

retinal and cortical implants

Mark Humayun with Connie Schoeman (RP)

Retina: Retina Implant AG, Second Sight, Boston U., and many othersCortex: Dobelle Institute (until 2004), CORTIVIS, U. Utah., IIT, U. NSW

“Jerry” with Dobelle brain implant Jens Naumann with Dobelle brain implant

www.jensnaumann.com

11%

4




7

non-invasive approaches

ubiquitous

cameras

mobile

computing

brain

plasticity

augmented

reality

computer

vision

13%


8

Robotic navigation aid for the blind?

SLAM (Simultaneous Localization and Mapping), USC on NTDTV July 4 2011

URL

James Weiland, USC Viterbi School of Engineering

16%

5




9

computer vision & vision technology

for the blind (news headlines in 2011)

- few technologies ever make it out of the lab- cost of commercialization, bulky, unreliable,physical limitations (Kinect fails in sunshine), …

EYE21- Universitat Politècnica de València, Spain

NAVI - Universität Konstanz, Germany

Kinect for the Blind - Moscow Institute of Physics

& Technology, Russia (MS Imagine Cup 2011)

MS Kinect

Stereo vision

More examples KinectFusion next?

18%


10

visual soundscapes

& augmented reality

Overview

• introduction





• conclusions

Overview

• introduction





• conclusions

cameraglasses

20%

6




11

sensory substitution

tactile or auditory display

Wicab’s BrainPort® The vOICe

ear buds

camera

tongue display

camera

23%


12

sensory substitution

with auditory display

Michael Proulx

25%

7




13

setups for The vOICe cost < $1000,

or < $500 when using a netbook PC

Michael Proulx

27%


14

visual soundscapes

& augmented reality

Overview

• introduction





• conclusions

Overview

• introduction





• conclusions

cameraglasses

30%

8




15

The vOICe image-to-sound mapping

Fre

quency

Fre

quency

TimeTime

Spectrographicsynthesis

32%


16

Could this really work?

Check: map sounds back to images

The vOICe image-to-sound mapping

Yes!(at least technically)

Theory

34%

9




17

The vOICe image reconstruction

“Soundscape”

37%


18

“Group performance differed statistically from chance level at all visual acuities up to 20/320””

“The visual acuity of the individual participants varied between 20/200 and 20/600”

“Interestingly, 5 of the 9 participants (55%) had visual acuity that exceeded the visual acuity threshold for blindness as defined by the WHO” (20/400)

perceptual resolution

beat 20/200 legal blindness barrier?

PLoS ONE 7(3), 2010, ``Visual’ Acuity of the Congenitally Blind Using Visual-to-Auditory

Sensory Substitution'', by Ella Striem-Amit, Miriam Guendelman and Amir Amedi::

39%

10




19

The vOICe mental imagery?

Warning:

Representationmay dramaticallyaffect recognition!

41%


20

The vOICe demo sounds (1 of 5)

Fre

quency

Fre

quency

TimeTime

44%

11




21


Fre

quency

Fre

quency

TimeTime

46%


22


Fre

quency

Fre

quency

TimeTime

48%

12




23


Fre

quency

Fre

quency

TimeTime

51%


24


Fre

quency

Fre

quency

TimeTime``Slow motion’’

Informational

masking? Apply

mental saccades

53%

13




25

Grasping exercise (can be mastered in a few hours)

55%


26

Grasping exercise (video clip from PLoS ONE 2008)Source: Proulx MJ, Stoerig P, Ludowig E, Knoll I, 2008 Seeing ‘Where’ through the Ears: Effects of Learning-by-Doing and Long-Term Sensory Deprivation on Localization Based on Image-to-Sound Substitution. PLoS ONE 3(3)

58%

14




27

[early blind] Metamodal LLC, California: NSF-fundedSBIR Phase I project with The vOICe, Pasadena 2010

http://www.youtube.com/playlist?list=PLDE409461FD43A4D060%


28

[late-blind] Metamodal LLC, California: NSF-fundedSBIR Phase I project with The vOICe, Pasadena 2010

http://www.youtube.com/playlist?list=PLDE409461FD43A4D062%

15




29

The vOICe

What do blind users see?

> In the top right hand corner of the picture is something which I

> recognise as architecture. It has 3 light coloured vertical columns

> below a sloping roof or sloping structure at the front above the

> columns. It is surrounded in shadow, especially on the right.

>

> Given that this is an architectural feature and not something smaller,

> that must mean that the vegetation on the top left of the picture is one

> or several trees.

> In the bottom left hand corner is 3 short dark columns and above it a V

> shaped band. These columns are shorter than the trees. To the right of

> the small columns are several short and long horizontal lines on a

> bright short band. I don't think they are columns but not sure what

> they are.

Photograph: courtesy Michael Larsen

65%


30

PF, late blind female, US, after hearing about visual cortex implants:

Just sound?.... No, It is by far more, it is sight ! There IS true light preception generated by the vOICe. When I am not wearing the voice the light I perceive from a small slit in my left eye is a grey fog. When wearing the vOICe the image is light with all the little greys and blacks. Yet a definite light image. True it is not color but it is definitely like looking at a black and

white TV show. The light generated is very white and clear then it erodes down

the scale of color to the dark black. I don't really see adiffrence in this light as compaired to the "light phosphenes " they are talking about.

The vOICe

Artificial synesthesia?

See also: Ward, J. and Meijer, P., ``Visual experiences in the blind induced by an auditory

sensory substitution device'', Consciousness and Cognition, Vol. 19, 2010, pp. 492–500.

67%

16




31

visual soundscapes

& augmented reality

Overview

• introduction




• conclusions

Overview

• introduction




• conclusions

69%


32

Neuroscience:Sound can activate “visual” cortex of blind

Kujala, Teija et al. (1995) “Visual cortex activation in blind humans during sound discrimination,” Neurosci. Lett. 183, 143–146

Early-blind person detecting pitch changes

72%

17




33

The vOICe in Neuroscience 2007:Cross-modal plasticity & visual sounds

Amedi et al., Nature Neuroscience, June 2007:

Shape conveyed by visual-to-auditory sensory substitutionactivates the lateral occipital complex

LOtv V1

dx.doi.org/10.1038/nn1912

Late-blind expert user of The vOICe(Talairach normalized inflated cortex reconstruction)

Percent signal change analysis of LOtv

LOtv acts as a metamodal operator for shape

74%


34

Harvard Medical School using The vOICe, Boston 2007

The Nature of Things

CBC television 2008

76%

18




35

Dennis, one of the recipients of a Dobelle brain implaint, complained during test stimulation of his visual cortexthat "the beeps were too loud"

This only happened when multiple electrodes were simultaneously stimulated. When individually stimulated,there were only visual sensations (phosphenes) for him

Dennis was 26 when he received the implant, after beingblind for 6 years

Dobelle brain implant

crossmodal perception?

Source: Jens Naumann, ``Search for paradise: A patient's account of the artificial vision

experiment'', www.jensnaumann.com, and personal communication

79%


36

visual soundscapes

& augmented reality

Overview

• introduction





• conclusions

Overview

• introduction





• conclusions

81%

19




37

The vOICe?

camera

earbuds

83%


38

augmented reality

camera view overlays

Smartphone-based

visual overlays

• Wikitude

• Layar

• …

For those without eyesight,

a sonic overlay

Layar Wikitude

86%

20




39

live soundscapes

talking locator (GPS)

talking compass

color identifier/filters

tactile feedback

foveal enlargement

The vOICe for Androidaugmented reality for the blind

voice commands multilingual (Russian, Spanish, Chinese, …)

barcode reader Google Goggles

88%


40

90%

21




41

The vOICe for Android

Google Analytics & mobile vision

Google Play

niche, yet in August 2011: 90,000+ total installs

4-week use: 135 countries

October 2012: 180,000+ total installs

93%


42

training & motivation…

http://www.nibib.nih.gov/Research/Highlights/Connectome

95%

22




43

visual soundscapes

& augmented reality

Overview

• introduction





• conclusions

Overview

• introduction





• conclusions

97%


44

conclusions:

� R&D ongoing

� affordable

� non-invasive

� available worldwide

http://http://www.seeingwithsound.comwww.seeingwithsound.com

visual soundscapes

& augmented reality

with thanks to all blind users and research partners around the world!100%

visual soundscapes from your augmented reality glasses& augmented reality overview •...

Documents