brain port vision device, mit kota(modi institute of technology)
DESCRIPTION
Modi Institute of Technology,kota (Kaushal)TRANSCRIPT
1
A
S e m i n a r r e p o r t o n“BRAINPORT VISION DEVICE”
Submitted toR a j a s t h a n T e c h n i c a l U n i v e r s i t y , K o t aIn Partial Fulfillment of the requirement for the award of the degree ofBachelor of Technology
In“Electronics & Communication Engineering”
2010-2011
Submitted by:- Supervised by:-
Koushal Singh Kiroula “Mr. Mansingh Negi”
(Dept. of Electronics & Communication Engg.)
M o d i I n s t i t u t e o f T e c h n o l o g y
N a y a g a o n , R a w a t b h a t a R o a d , K o t a
2
Chapter 1
INTRODUCTION
The BrainPort vision device provides information to blind individuals via a neuro-stimulating
array placed on the tongue. This unique device provides immediate benefits to its blind users
in areas of safety, mobility, and recreation and opens a new world of sensory experience and
exploration.
The BrainPort vision device operates by acquiring an image stream from a camera, similar to
a camcorder. Like a camcorder, the moving images are sent to a display, which, in this case,
is the electrode array on the tongue. The image stream is displayed on the tongue by
converting light information to electrical stimulation, which feels like microscopic bubbles to
the user. With minimal training, users learn to interpret the images on their tongue as
information about the scene in front of them.
The BrainPort vision device includes an imaging system capable of working both indoors and
outdoors, with a field of view spanning 3-75 degrees (magnified versus wide angle
views). The tongue array contains 400 electrodes and is connected to the controller via a
flexible cable. The control system is approximately the size of a PDA and runs for about 3
hours per charge, with swappable batteries.
An artist’s concept of the tongue array (or Intra-Oral Device, IOD) and camera mounting is
shown below alongside the device. The IOD is attached to a flexible boom by a thin
wire. The camera unit is mounted on a pair of eyeglasses frames. The user controls and the
power supply are connected to a belt-worn, pager-style, controller.
A blind woman sits in a chair holding a video camera focused on a scientist sitting in front of
her. She has a device in her mouth, touching her tongue, and there are wires running from
that device to the video camera. The woman has been blind since birth and doesn't really
know what a rubber ball looks like, but the scientist is holding one and when he suddenly
rolls it in her direction, she puts out a hand to stop it. The blind woman saw the ball through
her tongue. Well, not exactly through her tongue, but the device in her mouth sent visual
3
input through her tongue in much the same way that seeing individuals receive visual input
through the eyes. In both cases, the initial sensory input mechanism -- the tongue or the eyes -
- sends the visual data to the brain, where that data is processed and interpreted to form
images. Braille is a typical example of sensory substitution -- in this case, you're using one
sense, touch, to take in information normally intended for another sense, vision. Electro
tactile stimulation is a higher-tech method of receiving somewhat similar (although more
surprising) results, and it's based on the idea that the brain can interpret sensory information
even if it's not provided via the natural channel.
4
2.1 Statistics on the Blind:
37 million: People in the world are blind India (9 million), Africa (7 million) and
China (6 million)
Every 5 seconds: One person in our world goes blind
75 million: People will be blind by 2020
(if trends continue)
2.2 Cybernetics:
Cybernetics is about having a goal and taking action to achieve that goal.
"Cybernetics" comes from a Greek word meaning "the art of steering“.
Ironically but logically, AI and cybernetics have each gone in and out in the search for
machine intelligence
So “I Can Read” can be termed as a “Cybernetics System For Disabled (blind)”
5
Chapter 2
What is BrainPort Vision Device?
"BrainPort device" is a technology developed in US, which is making the world
visible to the ones who lose their sight due to some accidental incidents.
A Neuro scientist has developed the BrainPort Vision Device that allows the blinds to
“see” using their tongue.
Craig Lundberg, 24, is the first British soldier to test the BrainPort system, which is
billed as the next best thing to sight
The technology has made the dark-dependent world come alive and independent to
Craig Lundberg who completely lost his sight after a grenade attack in Iraq, as he is
now able to sense the visuals with his tongue. The soldier admits that his world has
been transformed because of the technology.
The device which sends visual input through tongue in much the same way that seeing
individuals receive visual input through the eyes is called the “BrainPort Vision
Device”.
BrainPort could provide vision-impaired people with limited forms of sight.
Technically, this device is underlying a principle called “electro tactile stimulation for
sensory substitution”.
To produce tactile vision, BrainPort uses a camera to capture visual data.
6
Chapter 3
Modes of stimulation and its various forms
From the very beginning of the electro tactile stimulation this journey has travelled a lot and
the various forms may be described as follows:
TVSS-Tactile Vision Substitution Systems.
Vibrotactile (FINGERTIPS).
Electro Tactile Stimulation for Tongue
3.1 Tactile vision substitution system
The TVSS may be characterized as a humanistic intelligence system. It represent a symbiosis
between instrumentation-for example, an artificial sensor array (TV camera)-computational
equipment, and the human user. Consistent with the terminology of this issue, this is made
possible by "instrumental sensory plasticity," the capacity of the brain to reorganize when
there is: (a) functional demand,(b) the sensor technology to fill that demand, and (c) the
training and psychosocial factors that support the functional demand. To constitute such
systems then, it is only necessary to present environmental information from an artificial
sensor in a form of energy that can be mediated by the receptors at the human-machine
interface, and for the brain, through a motor system (e.g., a head-mounted camera under the
motor control of the neck muscles), to determine the origin of the information.
This can now be extended into other domains with modern technology and the availability of
artificial sensory receptors, such as:
1. A miniature TV camera for blind persons,
2. A MEMS technology accelerometer for providing substitute vestibular information for
persons with bilateral vestibular loss,
3. Touch and shear-force sensors to provide information for spinal cord injured persons,
4. Instrumented condom for replacing lost sex sensation, or
5. A sensate robotic hand (Bach-y-Rita, 1999).
7
In first sensory substitution project, they developed tactile vision substitution Systems
(TVSS) to deliver visual information to the brain via arrays of stimulators in contact with the
skin of one of several parts of the body (abdomen, back thigh). Optical images picked up by a
TV camera were transuded into a form of energy (vibratory or direct electrical stimulation)
that could be mediated by the skin receptors. In these sensory substitute systems, the visual
information reaches the perceptual levels for analysis and interpretation via somatosensory
pathways and structures.
3.2 Vibrotactile
After sufficient training with the TVSS, our subjects reported experiencing the image in
space, instead of on the skin. They learn to make perceptual judgments using visual means of
analysis, such as perspective, parallax, looming and zooming, and depth judgments (Bach-y-
Rita, Collins, Saunders, White, & Scadden, 1969; cf., Bach-y-Rita, 1972, 1989, 1995, 1996,
1999; Bach-y-Rita, Kaczmarek, & Meier, 1998; Bach-y-Rita, Kaczmarek, Tyler, & Garcia-
Lara, 1998; Bach-y-Rita, Webster, Tompkins, & Crabb, 1987; Kaczmarek & Bach-y-Rita,
1995; White, Saunders, Scadden, Bach-y-Rita, & Collins, 1970). Although the TVSS systems
have only had between 100 and 1032 point arrays, the low resolution has been sufficient to
perform complex perception and "eye"-hand coordination tasks. These have included facial
recognition, accurate judgment of speed and direction of a rolling ball with over 95%
accuracy in batting a ball as it rolls over a table edge, and complex inspection-assembly tasks.
The latter were performed on an electronics company assembly line with a 100 point
vibrotactile array clipped to the work-bench against which the blind worker pressed the skin
of his abdomen, and through which information from a TV camera (Substituting for the
ocular piece of a dissection microscope) was delivered to the human-machine interface
(Bach-y-Rita, 1995, pp. 187-193).
8
Child reproducing perceived image of a teachers hand as displayed on a modified Optacon.
The tactile image is picked up with one finger statically placed on the 6 × 24 vibrotactile
array. LED monitor in foreground is visual representation of active pattern on the tactile
display, which is obtained by the child's head-mounted camera.
9
3.3 Electrotactile Stimulation for Tongue
In the TVSS studies cited above, the stimulus arrays presented only black-white information,
without gray scale. However, the tongue electrotactile system does present gray-scaled
pattern information, and multimodal and multidimensional stimulation is possible.
Simultaneously, we have also modelled the electrotactile stimulation parameter space to
determine how we might elicit tactile "colors."
Aiello (1998a, 1998b) has identified six stimulus parameters: the current level, the pulse
width, the interval between pulses, the number of pulses in a burst, the burst interval, and the
frame rate. All six parameters in the waveforms can, in principle, be varied independently
within certain ranges, and may elicit potentially distinct responses. For example, in a study of
electrical stimulation of the skin of the abdomen, Aiello (1998a) suggested that the best way
to encode intensity information independent of other percept qualities with a
multidimensional stimulus waveform was through modulation of the energy delivered by the
stimulus. In that case, the energy was varied in such a way that the displacement in the
parameter space, corresponding to a given transition between energy levels, was minimal
(gradient mode of stimulation). Although the gradient mode of stimulation requires a real-
time fulfilment of mathematical constraints among all the parameters, its implementation
could be included within a microelectronic package for signal treatment. The tongue interface
overcomes many of these. The tongue is very sensitive and highly mobile. Since it is in the
protected environment of the mouth, the sensory receptors are close to the surface. The
presence of an electrolytic solution, saliva, assures good electrical contact. The results
obtained with a small electrotactile array developed for a study of form perception with a
finger tip demonstrated that perception with electrical stimulation of the tongue is somewhat
better than with finger-tip electrotactile stimulation, and the tongue requires only about 3% of
the voltage (5-15 V), and much less current (0.4-2.0 mA), than the finger-tip. The electronic
system has been described elsewhere (Bach-y-Rita, Kaczmarek, Tyler, et al., 1998).
10
Electrotactile stimuli are delivered to the dorsum of the tongue via flexible electrode arrays
(Figure 3) placed in the mouth, with connection to the stimulator apparatus (TDU) via a flat
cable passing out of the mouth. The tongue electrode array and cable are made of a thin (100
μm) strip of polyester material (Mylar®) onto which a rectangular matrix of gold-plated
copper circular electrodes has been deposited by a photolithographic process similar to that
used to make printed circuit boards. The electrotactile stimulus consists of 40-μs pulses
delivered sequentially to each of the active electrodes in the pattern. Bursts of three pulses
each are delivered at a rate of 50 Hz with a 200 Hz pulse rate within a burst.
This structure was shown previously to yield strong, comfortable electrotactile precepts.
Positive pulses are used because they yield lower thresholds and a superior stimulus quality
on the fingertips.
Close-up of 144-point (12 x 12) "virtual ground" electrotactile tongue display
11
Action potentials (AP's) thus recorded had amplitudes from 0.1 to 1.0 mV and a 5 : 1 signal-
to-noise ratio (SNR).A circular electrode surrounding the recording site served as the ground
reference. Following pre amplification and band pass filtering (200-10 000 Hz), a differential
amplitude detector identified AP's, producing an output pulse whenever the recorded signal
entered a predefined amplitude-time window. In the first experiment, electrotactile
entrainment currents (iEN) were determined by adjusting the stimulation current from near
zero to the minimal value resulting in one AP for each stimulation pulse. These currents
exceeded the absolute thresholds (the currents causing occasional AP's) by approximately5%.
The entrainment current was determined twice for positive- and negative-polarity stimulation
pulses of ten different widths: 20, 30, 40, 50, 70, 100, 150, 200, 300, and 500 _s, delivered at
a rate of 10 pulses/s. The width sequence was reversed during the second run on each of the
three fibers.
Relative timing between simultaneous mechanical and electrotactile stimulation. The
top trace represents the sinusoidal, 30-Hz, 50-100-_m (0-P) mechanical displacement.
12
Chapter 4
Electrotactile Stimulation for Visual Substitution
When a human looks at an object, the optical image entering the eyes does not go beyond the
retina. Instead, it would turn into spatio-temporal nerve patterns of impulse along the optic
nerve fibers. By analysing the impulse patterns, the brain recreates the images. Indeed, the
channels such as eyes, ears and skin those carry sensory information to the brain are set up in
a similar manner to perform similar activities. To substitute one sensory input channel for
another, the big challenge to the scientists is how to correctly encode the nerve signals for the
sensory event and send them to the brain through the alternate channel as the brain appears to
the flexible when it comes to interpreting sensory input.
The concepts at work behind electro tactile stimulation for sensory substitution are complex.
The idea is to communicate non-tactile via electrical stimulation of the sense of touch.
In practice, this typically means that "an array of electrodes receiving input from a non-
tactile information source (a camera, for instance) applies small, controlled, painless currents
(some subjects report it feeling something like soda bubbles) to the skin at precise locations
according to an encoded pattern."
For a blind person, it means the encoding of the electrical Pattern essentially attempts to
mimic the input that would normally be received by the non-functioning sense – vision. So
patterns of light picked up by a camera to form an image are replacing the perception of the
eyes and converted into electrical pulses that represent those patterns of light. In other words,
when the encoded pulses are applied to the skin, the skin is actually receiving image data
which would be then sent to the brain in the forms of impulse. Under normal circumstances,
the parietal lobe in the brain receives touch information, while the occipital lobe receives
vision information. When the nerve fibers forward the image-encoded touch signals to the
13
parietal lobe, "the electric field thus generated in subcutaneous tissue directly excites the
afferent nerve fibres responsible for touch sensations".
Within the system, arrays of electrodes can be used to communicate non-touch information
through pathways to the brain normally used for the touch related impulses. The
breakthrough of the BrainPort technology is to use the tongue as the substitute sensory
channel.
14
Chapter 5
THE STRUCTURE OF THE BRAINPORT DEVICE
The figure below shows the structure of the BrainPort Vision Device.
The optical information that would normally hit the retina is picked up by the digital camera
in digital form. It uses radio signals to send the ones and zeros to the CPU for encoding. Each
set of pixels in the camera's light sensor corresponds to an electrode in the array. After that,
the CPU runs a program that turns the camera's electrical information into a spatially encoded
signal. "The encoded signal represents differences in pixel data as differences in pulse
characteristics such as frequency, amplitude and duration.
15
Multidimensional image information takes the form of variances in pulse current or voltage,
pulse duration, intervals between pulses and the number of pulses in a burst, among other
parameters." Then, the electrode array (shown in Fig) receives the resulting signal via the
stimulation circuitry and applies it to the tongue. At last, the brain interprets and uses the
information coming from the tongue as if it were coming from the eyes.
HOW DOES THE BRAINPORT VISION DEVICE WORK?
The BrainPort vision system consists of a postage-stamp-size electrode array for the top
surface of the tongue (the tongue array), a base unit, a digital video camera, and a hand-held
controller for zoom and contrast inversion. Visual information is collected from the user-
adjustable head-mounted camera (FOV range 3–90 degrees) and sent to the BrainPort base
unit. The base unit translates the visual information into an stimulation pattern that is
displayed on the tongue. The tactile image is created by presenting white pixels from the
camera as strong stimulation, black pixels as no stimulation, and gray levels as medium levels
of stimulation, with the ability to invert contrast when appropriate. Users often report the
sensation as pictures that are painted on the tongue with Champagne bubbles.
16
17
With the current system (arrays containing 100 to 600+ electrodes), study participants have
been able to recognize high-contrast objects, their location, movement, and some aspects of
perspective and depth. Trained blind participants use information from the tongue display to
augment understanding of the environment. Our ongoing research with the BrainPort vision
device demonstrates the great potential of tactile vision augmentation and we believe that
these findings warrant further exploration. As a result, we are currently working on
improvements to the tongue display hardware, software, and usability, and on overall device
miniaturization.
WHY TONGUE IS USED IN BRAINPORT TECHNOLOGY
Compare to all other skin areas, the tongue skin area is the most sensitive one because there
are more nerve fibres and they are much closer to the surface. Moreover, there is no stratum
conium (an outer layer of dead skin cells) on the tongue which act as an insulator. With these
characteristics, it requires less voltage to stimulate nerve fibres in the tongue (5 to 15 volts)
compared to areas like the fingertips or abdomen (40 to 500 volts). Also, since the tongue is
surrounded by saliva which contains electrolytes, it would help to maintain the current flow
between the electrode and skin tissues. Last but not least, the area of the cerebral cortex that
interprets touch data from the tongue is larger than the areas serving other body parts.
Therefore, the tongue is the best choice for conveying tactile-based data to the brain until this
moment
18
Chapter 6
ADVANTAGES:
Blind users can use the Brain Port vision device independently - at home, at work, and in
public spaces indoors and out as a tool for improved safety, mobility and object
recognition. Secondary benefits include applying the technology toward specific hobbies and
recreational situations. These benefits enable greater independence at home, school and in
business, greatly improving quality of life.
SAFETY
Navigating difficult environments, such as parking lots, traffic circles, complex
intersections
Recognizing quiet moving objects like hybrid cars or bicycle
MOBILITY
Finding doorways, hall intersections, lobby or restaurant in an office or hotel.
Finding continuous sidewalks, sidewalk intersections and curbs.
19
Chapter 7
OBJECT RECREATION
Locating people
Locating known objects such as shoes, cane, coffee mug, keys
APPLICATIONS
Just a few of the current or foreseeable applications include providing elements of
sight for the visually impaired in the medical field.
The BrainPort electrodes would receive input from a sonar device to provide not only
directional cues but also a visual sense of obstacles and terrain.
BrainPort may also provide expanded information for military pilots, such as a pulse
on the tongue to indicate approaching aircraft or to indicate that they must take
immediate action.
BrainPort applications include robotic surgery. The surgeon could wear electro tactile
gloves to receive tactile input from robotic probes inside someone's chest cavity.
20
21
Chapter 8
CURRENT AND POTENTIAL APPLICATIONS
While the full spectrum of BrainPort Vision Technology applications has yet to
realized, the device has the potential to lessen an array of sensory limitations and to
alleviate the symptoms of a variety of disorders.
Just a few of the current or foreseeable applications include providing elements of
sight for the visually impaired in the medical field.
Beyond medical applications, scientists have been exploring potential military uses
with a grant from the Defence Advanced Research Projects Agency (DARPA). They
are looking into underwater applications that could provide the Navy Seals with
navigation information and orientation signals in dark, murky water.
The BrainPort electrodes would receive input from a sonar device to provide not only
directional cues but also a visual sense of obstacles and terrain.
BrainPort may also provide expanded information for military pilots, such as a pulse
on the tongue to indicate approaching aircraft or to indicate that they must take
immediate action.
Other potential BrainPort applications include robotic surgery. The surgeon could
wear electro tactile gloves to receive tactile input from robotic probes inside
someone's chest cavity. In this way, the surgeon could feel what he's doing as he
controls the robotic equipment.
Race car drivers might use a version of BrainPort to train brains for faster reaction
times, and gamers might use electro tactile feedback gloves or their controllers to feel
what they're doing in a video game.
22
Chapter 9
CONCLUSION
Even though this is a field of scientific study that has been around for nearly a century, it has
just picked up in this decade due to the miniaturization of electronics and dramatic
improvement of the computer's speed. Already more streamlined than any previous setup
using electro tactile stimulation for sensory substitution, BrainPort envisions itself even
smaller and less obtrusive in the future. In the case of the BrainPort Vision Device the
electronics might be completely embedded in a pair of glasses along with a tiny camera and
radio transmitter, and the mouthpiece would house a radio receiver to receive encoded signals
from the glasses. It's not exactly a system on a chip, but gives it 20 years -- we might be
seeing a camera with the size of a grain of rice embedded in people's foreheads.
23
REFERENCES
Bach-y-Rita, Paul et al. "Form perception with a 49-point electro tactile stimulus
array on the tongue: A technical note." Journal of Rehabilitation Research and
Development.
http://howstuffworks.com
http://india.cchem.berkeley.edu/~reimer/courses/present/samples/sample2.pdf
http://wistechnology.com/printarticle.php?id=2319
http://www.itechnews.net/2009/08/16/brainport-vision-device-lets-blinds-see-with-
tougues/
http://www.gizmowatch.com/entry/brainport-vision-device-for-the-visually-
impaired-to-feel-images/
http://accessibletravel.suite101.com/article.cfm/brainport_vision_device
http://www.scientificamerican.com/article.cfm?id=device-lets-blind-see-with-tongues
http://topnews.net.nz/content/22766-brainport-device-transforms-world-blinds
http://www.telegraph.co.uk/health/healthnews/7350218/British-soldier-blinded-in-
Iraq-trials-new-technology-to-see-using-his-tongue.html