Visual soundscapes from your augmented reality glasses

CONGres 2012 - Visual soundscapes from your augmented reality glasses

visual soundscapes
from your augmented reality glasses

CONGres 2012
October 17 2012, Amsterdam

Sounds in Science

Peter Meijer, Metamodal BV
www.seeingwithsound.com

© 2012 Peter B.L. Meijer

0%

2
camera
display

you in 2014?


2%

E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 1


3

?

4%

4

visual soundscapes
& augmented reality

Overview
Overview
•• introduction
introduction
•• sensory substitution
sensory substitution
camera
glasses •• visual soundscapes
visual soundscapes
•• brain plasticity
brain plasticity
•• augmented reality
augmented reality
•• conclusions
conclusions

6%



5

visual soundscapes
& augmented reality

Overview
Overview
•• introduction
introduction
camera
visual soundscapes
brain plasticity
augmented reality
•• conclusions
conclusions

9%

6

bionic eye
retinal and cortical implants
www.jensnaumann.com

“Jerry” with Dobelle brain implant Jens Naumann with Dobelle brain implant
Mark Humayun with Connie Schoeman (RP)

Retina: Retina Implant AG, Second Sight, Boston U., and many others
Cortex: Dobelle Institute (until 2004), CORTIVIS, U. Utah., IIT, U. NSW © 2012 Peter B.L. Meijer

11%



7

non-invasive approaches
computer
mobile brain
vision
computing plasticity

ubiquitous augmented
cameras reality


13%

8

Robotic navigation aid for the blind?
James Weiland, USC Viterbi School of Engineering

URL

SLAM (Simultaneous Localization and Mapping), USC on NTDTV July 4 2011 © 2012 Peter B.L. Meijer

16%



9

computer vision & vision technology
for the blind (news headlines in 2011)
More examples MS Kinect KinectFusion next?

Stereo vision
NAVI - Universität Konstanz, Germany
EYE21- Universitat Politècnica de València, Spain
Kinect for the Blind - Moscow Institute of Physics
& Technology, Russia (MS Imagine Cup 2011)

- few technologies ever make it out of the lab
- cost of commercialization, bulky, unreliable,
physical limitations (Kinect fails in sunshine), …

18%

10

visual soundscapes
& augmented reality

Overview
Overview
•• introduction
introduction
camera
visual soundscapes
brain plasticity
augmented reality
•• conclusions
conclusions

20%



11

tactile or auditory display
camera

ear buds

camera

tongue
display

Wicab’s BrainPort® The vOICe © 2012 Peter B.L. Meijer

23%

12

with auditory display

Michael Proulx


25%



13

setups for The vOICe cost < $1000,
or < $500 when using a netbook PC

Michael Proulx


27%

14

visual soundscapes
& augmented reality

Overview
Overview
•• introduction
introduction
camera
visual soundscapes
brain plasticity
augmented reality
•• conclusions
conclusions

30%



15

The vOICe image-to-sound mapping
Frequency

Spectrographic
synthesis Time

32%

16

The vOICe image-to-sound mapping

Could this really work?

Theory Yes!
(at least technically)

Check: map sounds back to images

34%



17

The vOICe image reconstruction

“Soundscape”


37%

18

perceptual resolution
beat 20/200 legal blindness barrier?
PLoS ONE 7(3), 2010, ``Visual’ Acuity of the Congenitally Blind Using Visual-to-Auditory
Sensory Substitution'', by Ella Striem-Amit, Miriam Guendelman and Amir Amedi: :

“Group performance differed statistically from chance level
at all visual acuities up to 20/320”
”

“The visual acuity of the individual participants varied
between 20/200 and 20/600”

“Interestingly, 5 of the 9 participants (55%) had visual acuity
that exceeded the visual acuity threshold for blindness as
defined by the WHO” (20/400)

39%



19

The vOICe mental imagery?

Warning:

Representation
may dramatically
affect recognition!


41%

20

The vOICe demo sounds (1 of 5)
Frequency

Time

44%



21

Frequency

Time

46%

22

Frequency

Time

48%



23

Frequency

Time

51%

24


Informational
Frequency

masking? Apply
mental saccades

Time
``Slow motion’’ © 2012 Peter B.L. Meijer

53%



25

Grasping exercise (can be mastered in a few hours) © 2012 Peter B.L. Meijer

55%

26

Grasping exercise (video clip from PLoS ONE 2008)
Source: Proulx MJ, Stoerig P, Ludowig E, Knoll I, 2008 Seeing ‘Where’ through the Ears: Effects of Learning-by-
Doing and Long-Term Sensory Deprivation on Localization Based on Image-to-Sound Substitution. PLoS ONE B.L. Meijer
© 2012 Peter 3(3)

58%



27

[early blind] Metamodal LLC, California: NSF-funded
SBIR Phase I project with The vOICe, Pasadena 2010 © 2012 Peter B.L. Meijer

60% http://www.youtube.com/playlist?list=PLDE409461FD43A4D0

28

[late-blind] Metamodal LLC, California: NSF-funded
SBIR Phase I project with The vOICe, Pasadena 2010 © 2012 Peter B.L. Meijer

http://www.youtube.com/playlist?list=PLDE409461FD43A4D0
62%



29

The vOICe
What do blind users see?
> In the bottom left hand corner is 3 short dark columns and above it a V
> shaped band. These columns are shorter than the trees. To the right of
> the small columns are several short and long horizontal lines on a
> bright short band. I don't think they are columns but not sure what
> they are.

> In the top right hand corner of the picture is something which I
> recognise as architecture. It has 3 light coloured vertical columns
> below a sloping roof or sloping structure at the front above the
> columns. It is surrounded in shadow, especially on the right.
>
> Given that this is an architectural feature and not something smaller,
Photograph: courtesy Michael Larsen
> that must mean that the vegetation on the top left of the picture is one
> or several trees. © 2012 Peter B.L. Meijer

65%

30

The vOICe
Artificial synesthesia?

PF, late blind female, US, after hearing about visual cortex implants:
Just sound?.... No, It is by far more, it is sight ! There IS true light
preception generated by the vOICe. When I am not wearing the voice the
light I perceive from a small slit in my left eye is a grey fog. When wearing the
vOICe the image is light with all the little greys and blacks. Yet a definite
light image. True it is not color but it is definitely like looking at a black and
white TV show. The light generated is very white and clear then it erodes down
the scale of color to the dark black. I don't really see adiffrence in this light
as compaired to the "light phosphenes " they are talking about.

See also: Ward, J. and Meijer, P., ``Visual experiences in the blind induced by an auditory
sensory substitution device'', Consciousness and Cognition, Vol. 19, 2010, pp. 492–500.

67%



31

visual soundscapes
& augmented reality

Overview
Overview
•• introduction
introduction
brain plasticity
augmented reality
•• conclusions
conclusions


69%

32

Neuroscience:
Sound can activate “visual” cortex of blind
Kujala, Teija et al. (1995) “Visual cortex activation in blind humans during sound discrimination,” Neurosci. Lett. 183, 143–146

Early-blind person detecting pitch changes © 2012 Peter B.L. Meijer

72%



33

The vOICe in Neuroscience 2007:
Cross-modal plasticity & visual sounds
dx.doi.org/10.1038/nn1912
Amedi et al., Nature Neuroscience, June 2007:
Shape conveyed by visual-to-auditory sensory substitution
activates the lateral occipital complex
LOtv acts as a metamodal operator for shape
Percent signal change analysis of LOtv

LOtv V1

Late-blind expert user of The vOICe
(Talairach normalized inflated cortex reconstruction)


74%

34

The Nature of Things
CBC television 2008

Harvard Medical School using The vOICe, Boston 2007 © 2012 Peter B.L. Meijer

76%



35

Dobelle brain implant
crossmodal perception?
Dennis, one of the recipients of a Dobelle brain implaint,
complained during test stimulation of his visual cortex
that "the beeps were too loud"

This only happened when multiple electrodes were
simultaneously stimulated. When individually stimulated,
there were only visual sensations (phosphenes) for him

Dennis was 26 when he received the implant, after being
blind for 6 years
Source: Jens Naumann, ``Search for paradise: A patient's account of the artificial vision
experiment'', www.jensnaumann.com, and personal communication

79%

36

visual soundscapes
& augmented reality

Overview
Overview
•• introduction
introduction
•• visual soundscapes
visual soundscapes
brain plasticity
augmented reality
•• conclusions
conclusions

81%



37
camera
earbuds

The vOICe?


83%

38

augmented reality
camera view overlays
Layar Wikitude
Smartphone-based
visual overlays
• Wikitude
• Layar
• …

For those without eyesight,
a sonic overlay

86%



39

The vOICe for Android
augmented reality for the blind
live soundscapes barcode reader Google Goggles

foveal enlargement

talking locator (GPS)

talking compass

color identifier/filters

tactile feedback

voice commands multilingual (Russian, Spanish, Chinese, …)

88%

40


90%



41

The vOICe for Android
Google Analytics & mobile vision

niche, yet in August 2011: 90,000+ total installs
4-week use: 135 countries
October 2012: 180,000+ total installs

Google Play


93%

http://www.nibib.nih.gov/Research/Highlights/Connectome 42

training & motivation…

95%



43

visual soundscapes
& augmented reality

Overview
Overview
•• introduction
introduction
•• visual soundscapes
visual soundscapes
brain plasticity
augmented reality
•• conclusions
conclusions

97%

44

visual soundscapes
& augmented reality

conclusions:
R&D ongoing
affordable
non-invasive
available worldwide

http://www.seeingwithsound.com
http://www.seeingwithsound.com

with thanks to all blind users and research partners around the world!

100%


Visual soundscapes from your augmented reality glasses

Recommandé

Recommandé

Contenu connexe

Similaire à Visual soundscapes from your augmented reality glasses

Similaire à Visual soundscapes from your augmented reality glasses (6)

Dernier

Dernier (20)

Visual soundscapes from your augmented reality glasses