camera-based sensory substitution and augmented reality for the blind (acivs 2011)

1

ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind

E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com

© 2011 Peter B.L. Meijer

ACIVS 2011August 22-25, Ghent, Belgium

Peter Meijer

Metamodal BVThe Netherlands

www.artificialvision.com

Camera-based sensory substitution

and augmented reality for the blind

0%


2



Overview

• introduction

• sensory substitution

synesthesia & philosophy

• brain plasticity

• augmented reality

• conclusions

Overview

• introduction





• conclusions

cameraglasses

2%

2




3



Overview

• introduction





• conclusions

Overview

• introduction





• conclusions

4%


4

Computer vision & vision technology

for the blind

Computer vision, thinking of…

- contrast enhancement, dehazing (cataracts) - color identification / filtering

- edge detection / enhancement (low acuity vision)

- OCR (Optical Character Recognition)- object detection / recognition (SIFT, SURF, …)

- collision threat analysis (e.g. optic flow)- saliency detection / enhancement - SLAM (Simultaneous Localization and Mapping)

6%

3




5


for the blind

Goals

- improve blind orientation and mobility (O&M)through obstacle detection, navigation aids,object recognition (also crowdsourcing), …

- offer “sight” through sensory substitution(convey live camera views by sound or touch)

[also hybrid approaches and augmented reality]

9%


6

Electronic glasses…

Image enhancement visually impaired

Bristol Mobility Aid - University of the West of England, Bristol, UKMobility performance of low-vision adults using an electronic mobility aid, Jones et al., Clinical and Experimental Optometry 2006 89:1 pp 10-17

11%

4




7

Robotic navigation aid for the blind?

SLAM (Simultaneous Localization and Mapping), USC on NTDTV July 4 2011

URL

James Weiland, USC Viterbi School of Engineering

13%


8


for the blind (news headlines in 2011)

- few technologies ever make it out of the lab- cost of commercialization, bulky, unreliable,physical limitations (Kinect fails in sunshine), …

EYE21- Universitat Politècnica de València, Spain

NAVI - Universität Konstanz, Germany

Kinect for the Blind - Moscow Institute of Physics

& Technology, Russia (MS Imagine Cup 2011)

MS Kinect

Stereo vision

More examples KinectFusion next?

15%

5




9

How to give sight to the blind?

ubiquitous

cameras

mobile

computing

brain

plasticity

augmented

reality

computer

vision

when conventional

medical approaches

do not suffice…

18%


10


The vOICe @ APA2011Michael Proulx

Preview

20%

6




11

gene therapy

retinal implants

cortical implants

LGN implants

sensory substitution


when conventional

medical approaches

do not suffice

22%


12



Overview

• introduction





• conclusions

Overview

• introduction





• conclusions

cameraglasses

25%

7




13

Sensory substitution

Tactile or auditory display

Wicab’s BrainPort® The vOICe

ear buds

camera

tongue display

camera

27%


14

The vOICe image-to-sound mapping

Fre

quency

Fre

quency

TimeTime

Spectrographicsynthesis

29%

8




15

Could this really work?

Check: map sounds back to images

The vOICe image-to-sound mapping

Yes!(at least technically)

Theory

31%


16

The vOICe image reconstruction

“Soundscape”

34%

9




17

The vOICe mental imagery?

Warning:

Representationmay dramaticallyaffect recognition!

36%


18

The vOICe demo sounds (1 of 5)

Fre

quency

Fre

quency

TimeTime

38%

10




19


Fre

quency

Fre

quency

TimeTime

40%


20


Fre

quency

Fre

quency

TimeTime

43%

11




21


Fre

quency

Fre

quency

TimeTime

45%


22


Fre

quency

Fre

quency

TimeTime``Slow motion’’

Informational

masking? Apply

mental saccades

47%

12




23

Setups for The vOICe cost < $1000,

or < $500 when using a netbook PC

Michael Proulx

50%


24

Grasping exercise (can be mastered in a few hours)

52%

13




25

Grasping exercise (video clip from PLoS ONE 2008)Source: Proulx MJ, Stoerig P, Ludowig E, Knoll I, 2008 Seeing ‘Where’ through the Ears: Effects of Learning-by-Doing and Long-Term Sensory Deprivation on Localization Based on Image-to-Sound Substitution. PLoS ONE 3(3)

54%


26

[early blind] Metamodal LLC, California: NSF-fundedSBIR Phase I project with The vOICe, Pasadena 2010

http://www.youtube.com/playlist?list=PLDE409461FD43A4D056%

14




27

[late-blind] Metamodal LLC, California: NSF-fundedSBIR Phase I project with The vOICe, Pasadena 2010

59% http://www.youtube.com/playlist?list=PLDE409461FD43A4D0


28

[late-blind] Wicab BrainPort® vision device, Pittsburgh 2011

Benchmarking?

61%

15




29

The vOICe

What do blind users see?

> In the top right hand corner of the picture is something which I

> recognise as architecture. It has 3 light coloured vertical columns

> below a sloping roof or sloping structure at the front above the

> columns. It is surrounded in shadow, especially on the right.

>

> Given that this is an architectural feature and not something smaller,

> that must mean that the vegetation on the top left of the picture is one

> or several trees.

> In the bottom left hand corner is 3 short dark columns and above it a V

> shaped band. These columns are shorter than the trees. To the right of

> the small columns are several short and long horizontal lines on a

> bright short band. I don't think they are columns but not sure what

> they are.

Photograph: courtesy Michael Larsen

63%


30



Overview

• introduction





• conclusions

Overview

• introduction





• conclusions

65%

16




31

PF, late blind female, US, after hearing about visual cortex implants:

Just sound?.... No, It is by far more, it is sight ! There IS true light preception generated by the vOICe. When I am not wearing the voice the light I perceive from a small slit in my left eye is a grey fog. When wearing the vOICe the image is light with all the little greys and blacks. Yet a definite light image. True it is not color but it is definitely like looking at a black and

white TV show. The light generated is very white and clear then it erodes down

the scale of color to the dark black. I don't really see adiffrence in this light as compaired to the "light phosphenes " they are talking about.

The vOICe

Artificial synesthesia?

See also: Ward, J. and Meijer, P., ``Visual experiences in the blind induced by an auditory

sensory substitution device'', Consciousness and Cognition, Vol. 19, 2010, pp. 492–500.

68%


32

Philosophy and The vOICe

What does

that mean?

What does

that mean?

17th Century 17th Century ““MolyneuxMolyneux problemproblem”” revisited?revisited?

•• Q:Q: Would someoneWould someone born blind, born blind, in case sight was restored,in case sight was restored,

be able to be able to tell a cube from a sphere by sighttell a cube from a sphere by sight alonealone??

A:A: ““NoNo”” ((according to John Locke & William according to John Locke & William MolyneuxMolyneux))

•• Q:Q: Would someoneWould someone born blind, born blind, using using ““seeingseeing--withwith--soundsound””,,

be able to be able to tell a cube from a sphere by tell a cube from a sphere by ““sightsight”” alonealone??

A:A: ““YesYes””

70%

17




33



Overview

• introduction





• conclusions

Overview

• introduction





• conclusions

72%


34

Neuroscience:Sound can activate “visual” cortex of blind

Kujala, Teija et al. (1995) “Visual cortex activation in blind humans during sound discrimination,” Neurosci. Lett. 183, 143–146

Early-blind person detecting pitch changes

75%

18




35

The vOICe in Neuroscience 2007:Cross-modal plasticity & visual sounds

Amedi et al., Nature Neuroscience, June 2007:

Shape conveyed by visual-to-auditory sensory substitutionactivates the lateral occipital complex

LOtv V1

dx.doi.org/10.1038/nn1912

Late-blind expert user of The vOICe(Talairach normalized inflated cortex reconstruction)

Percent signal change analysis of LOtv

LOtv acts as a metamodal operator for shape

77%


36

Harvard Medical School using The vOICe, Boston 2007

The Nature of Things

CBC television 2008

79%

19




37



Overview

• introduction





• conclusions

Overview

• introduction





• conclusions

81%


38

Augmented Reality

Camera view overlays

Smartphone-based

visual overlays

• Wikitude

• Layar

• …

For those without eyesight,

a sonic overlay: The vOICe

Layar Wikitude

84%

20




39

86%


40

live soundscapes

talking locator (GPS)

talking compass

color identifier/filters

tactile feedback

foveal enlargement

The vOICe for Androidaugmented reality for the blind

voice commands multilingual (Russian, Spanish, Chinese, …)

barcode reader Google Goggles

88%

21




41

The vOICe for Android

Google Analytics & mobile vision

Android Market

niche, yet in August 2011: 90,000+ total installs

4-week use: 135 countries

90%


42

Augmented Reality

Hands-free future?

Ralph Osterhout, EG 2011

93%

22




43

Netbook + $30 USB camera glasses + optional $20 wide-angle lens

Back to reality: what blind users of The vOICe are using today

95%


44



Overview

• introduction





• conclusions

Overview

• introduction





• conclusions

97%

23




45

Conclusions:

� Research ongoing

� Affordable

� Non-invasive

� Available worldwide

http://http://www.artificialvision.comwww.artificialvision.com



with thanks to all blind users and research partners around the world!100%