THE MAN WHO MISTOOK HISCOMPUTER FOR A HAND

Neural Control of Robotic Devices

ELIE BIENENSTOCK, MICHAEL J. BLACK, JOHN DONOGHUE, YUN GAO, MIJAIL SERRUYA

BROWN UNIVERSITY

Applied Mathematics Computer Science Neuroscience

XEROX ALTO 1973

Dell workstation, 2001

BRAIN-COMPUTER INTERFACES

“If I could find … a code which translates the relation between the reading of the encephalograph and the mental image …the brain could communicate with me.”

Curt Siodmak, 1942

Brain

“Mad” scientist Nancy Reagan

NEURAL PROSTHETICS

Sensation

Action

NEURAL PROSTHETICS

Sensation

Action

NEURAL PROSTHETICS

Robot control?

Representation, inference, and algorithms?

Signals?

< 200 ms

VISUALIZING THE PLAYERS

Single cells ofthe nervous system

NEURON

BRAIN VERSUS COMPUTER

50,000,000Transistors (P IV)

Computational Elements

100,000,000,000Neurons

Speed (operations/second/element)30-300 1.5 * 109

MASSIVE CONNECTIVITY

SYNAPSES

Conturo et al., 1999

LONG DISTANCE CALLS

THE EEG

SINGLE UNIT ACTIVITY

1/10 mm

2/1000’s second

Spikes

CELL ENSEBLES

B

PMA

CentralSulcus

Arcuate

SMAMIMI

5 mm

Implanted in the MI arm area of motor cortex- lacks strict somatopy

PMA

1 ms

80µV

100 electrodes, 400µm separation4x4 mm

Neural ImplantNeural Implant

500 µm

Bone

Connector

Silicone

Dura

White Matter

400 µm

Cortex

I

III

V

VI

Arachnoid space

Connector Acyrlic

Chronically implanted.Stable recording for 2-3 years

(but not necessarily the same cells every day)

PATTERNS IN NEURAL ACTIVITY

LANGUAGE OF THE BRAIN

Language of the brain. Language of the computer.

Interpretation

“Translation”

AMBIGUOUS SIGNALS

INFERENCE

Ambiguous measurements

?Inference

“Prior” knowledge about how brains work.

“Prior” knowledge about the environment

History of measurements and interpretations

GOALS

* Model neural activity in motor cortex.* Explore how the brain codes information.* Model the statistical relationship between

neural activity and action.* Develop new statistical methods for

analyzing neural codes.* Build prosthetic devices to assist the severely

disabled.* Explore new output devices that can be controlled

by brains.

MODELING NEURAL FUNCTION

Simultaneously record hand position, velocity, and neural activity in motor cortex.

ARM MOTION

Distribution of training motions:

Position Velocity

MODELING NEURAL FUNCTION

direction

speed0

3

0

3

time…

direction θ (radians)

spee

d, r

, (cm

/s)

Cell 3

Spik

e tra

ins

MODELING NEURAL FUNCTION

direction

speed0

3

0

3

time…

direction θ (radians)

spee

d, r

, (cm

/s)

Cell 3

Spik

e tra

ins

NEURAL ACTIVITY

−π πdirection θ (radians)

0

12

spee

d, r

, (cm

/s)

Cell 3 Cell 16 Cell 19

Is there some “true” underlying response function?

Infer from using Bayesian inference.

NON-PARAMETRIC MODEL

vf

Tr ],[ θ=v

vg

vf

vg

: Observed mean firing rate for velocity v

: True mean firing rate for velocity v

is a noisy realization of the model

vf

vg

likelihood spatial prior

))|()|(()|( 1 ivvivv ggpgfpp ηκ =∏∏= vfg

LIKELIHOOD

gfP eg

fgfp −=

!1

)|(

Observed firing rate modeled a sample from

Poisson:

SPATIAL PRIOR

Markov Random Field assumption

∏=

=η

1

)|()|(i

vvv iggpgp g

T-distribution.

222

3

)(2

)(g

gp∆+

=∆σπ

σ

OOPTIMIZATION

∏=v

vvv gpgfpp )|()|()|( gfg κ

Many ways to maximize over gv

• Simulated annealing, etc.• We exploit an approximate

deterministic regularization method.• Take the negative log of• Minimize using gradient descent• Not ideal (loopy propagation, see Yedidia, Freeman, & Weiss, NIPS’00).

)|( fgp

Poisson+RobustPoisson+Robust

−π πangle (radians)

0

12

Cell 3 Cell 16 Cell 19

INFERENCE FROM ACTIVITY

speed

?direction

…

t

Inference

PredictionNeural Data

INFERENCE FROM ACTIVITY

speed

?direction

…

t

“prior” knowledge

PredictionNeural Data

GOALS

sound probabilistic inference

causal

estimate over short time intervals to reduce lag

cope with non-linear dynamics of hand motion

cope with ambiguities (multi-modal distributions)

more realistic firing models (Poisson or Poisson+refractory period [Kass&Venture, Neural Comp. ’01])

support higher level analysis of activities

likelihood

∏=

−=cellsi

gc

titt

tti

teg

cp s

ssc ,

!1

)|(,

…

)|()|()|( 12 −= tttttt ppp CsscCs κ

BAYESIAN INFERENCE

Want to infer state of hand given the activity, Ct, of (~25 cells) up to time t.

],[ θrt =s

prior

∫ −−−−− = 11111 )|()|()|( ttttttt dppp sCsssCs

Temporal dynamics(constant velocity)

)|()|()|( 12 −= tttttt ppp CsscCs κ

BAYESIAN INFERENCE

PARTICLE FILTER

samplesample

samplesample

normalizenormalize

Posterior)|( 11 −− ttp Cs

Temporal dynamics)|( 1−ttp ss

Likelihood)|( ttp sc

Posterior)|( ttp Cs

Represent posterior with a discrete set of N states and their normalized likelihood.

PARTICLE FILTER

Isard & Blake ‘96

1000 “S1000 “SYNTHETICYNTHETIC” C” CELLSELLS

Particle filtering in 50 ms:

vx: r2 = 0.8746

vy: r2 = 0.9033

A NEURAL PROSTHETIC

EEG

single and multi-neuronactivity

Voluntary control signal

mathematical algorithm

Computer cursorand

keyboard entryRobotic arm

Stimulation of Muscles,Spinal Cord, and Brain

NEURAL-PROSTHETIC LIMBS

HYBRID SYSTEMS

Motor control

Sensory input

Connecting Brains to Robots, Reger et al, Artificial Life, 2000.

BRAIN/MACHINE HYBRIDS

Explore biological sensory/control systems with artificial systems.

Develop computational models of biological control.

Re-map input modalities.

Opportunity for robotic prostheses.

Augment limited neural control with autonomy (eg. obstacle avoidance).

OUR BODIES OURSELVES?

Probotics/Jim Judkis

Service robots under neural control.

Sensation and action at a distance.

Stimulating the brain.

Ethics, liminality, fear, and the “uncanny”.

Honda robot

THANKS

D. Sheinberg, NeuroscienceN. Hatsopoulos, NeuroscienceW. Patterson, EngineeringA. Nurmikko, EngineeringG. Friehs, Brown Medical SchoolS. Geman, Division of Applied MathematicsM. Fellows, NeuroscienceL. Paninski, NYU, Center for Neural ScienceN.K. Logothetis, Max Planck Institute, Tuebingen