Cortical Control of Reaching and Grasping: Basic Science and Applications to

Brain-Machine Interfaces

Courtesy of Maryam Saleh

Postural Stabilization & Foraging

Locomotion

Food Handling

Gesturing & Communication

Tool use

Social interactions

• paths are relatively straight• bell-shaped velocity profile

Point-to-point reaching

Hollerbach & Flash (1982)

Hollerbach & Flash (1982)

TAN

GEN

TIA

L VE

LOC

ITY

(CM

/S)

NO

RM

ALI

ZED

TA

NG

ENTI

AL

VELO

CIT

Y

Atkeson & Hollberbach (1985)

Reaching in the sagittal plane

Dorsal reach network

IPL

PRR

VI V2 MI PFC

Giant Betz cellsMI: agranular cortex

Directional Tuning in Primary Motor Cortex

time (s)

0°

45°90°

135°

180°

225°270°

315°

20 40

frequency (Hz)150

0-0.5 1.00

Population Vector Decoding

Estimated DirectionΣ

Weighted Vector Sum

Cosine Directional tuning

0 100 200 3000

50

100

direction

firin

g ra

te (H

z)

i

N

ii PDf∑

=1

Ada

pted

from

Geo

rgop

oulo

s, A

.P.,

Neu

roph

ysio

logy

of R

each

ing,

in Je

anne

rod,

M. (

Ed.),

Atte

ntio

n an

d Pe

rfor

man

ce X

III: M

otor

Rep

rese

ntat

ion

and

Con

trol.

Hill

sdal

e, N

J: L

awre

nce

Erlb

aum

Ass

ocia

tes,

1990

, pp

. 227

–263

Kakei et al. (2001)

PMv:

muscle-centeredor body-centered?

primarily body-centered

E-ExtensionF-FlexionU-Ulnar deviationR-Radial deviation

go cue

Ventral grasp network

IPL

PRR

Grasp

Napier (1956)

Power grip Precision grip

Napier (1956)

Santello et al. (1998)

Grasp postures to imagined objects: Eigenpostures

No clustering of power and precision grips

AIP (Anterior Intra-Parietal Cortex)

Murata et al. (2000)

Multidimensional scaling: AIP responds to object shape

Murata et al. (2000)

2 “Motor” cells

Two types of PMv neurons: “Motor” & “Visuomotor”

Raos et al. (2006)

Raos et al. (2006)

2 “Visuomotor” cells

2 “Visuomotor” cells with orientation selectivity

Raos et al. (2006)

Clustering analysis: PMv responds to grip type instead of object shape

Raos et al. (2006)

Coordination of reach and grasp

transport

aperture

θ

orientation

•It is a complex movement that turns visual information about the object into a motor command.

•Involves coordinating proximal arm muscles with individuated finger movements to grasp an object

•It is an ethologically relevant behavior

•Few neuroscience studies have investigated the coordination of reaching and grasping.

Coordination of Reach and Grasp

Spatial Coordination

Temporal Coordination

Speed ApertureTime to

Peak deceleration vs Maximum Aperture

Correlation coefficient 0.78, p<0.01

f(x) = 0.04 + 0.77x

Mind/Brain Bodycorticospinal tract

proprioception

Applications to Brain-Machine Interfaces

Can we recover the link from the mind to the body when the link is broken?

Answer: yes, using a Neuro-motorProsthetic system

Computer

Neuro-motor prosthetic system

1) Multi-electrode array implant

2) Decoding of neural signals

3) Output interface

Personal computer:• mouse• keyboard

Assistive Robotics:• robotic arm• mechanized prosthetic arm

Biological interface:• muscles • peripheral nerve • spinal cord

Utah/Bionic Technologies ProbeRichard Normann U Utah

LegLeg

FaceFace

ArmArm

5 mm5 mm

PMdPMd

MIMI

Histology

400µm

Array Insertion sites

GFAP immunohistochemistryNissl Stain- 40 mm parasagittal(Thionine)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0 100 200 300 400 500 600 700 800 900 1000

DAYS POSTIMPLANT

AR

RA

Y YI

ELD

subject 1 (tom)subject 2 (coco)subject 3 (buddy)subject 4 (radley)

Long-term Reliability & Stability

( ) XRRRf T1T −=

Warland et al. (1997)

Decoding Algorithm:Optimal Linear Filter Reconstruction

RfX(t)=ˆ

Estimated position of handin time

Response of neural ensemble in time

filter coefficients

Neural Activity (firing rate f)

Overt Behavior(position x,y)

Map RelationshipRf = x,y

In absence of overt behavior

Rf = ?

How do you build a brain-machine interface decoder with a quadiplegicpatient who can’t move?

Does the visual display of action automatically trigger movement commands?

x

y

x

y

Experiment Conditions

Action Observation

Action

Action

Action Observation

Peri-target hit time histograms

Tuning to Cursor Velocity: Mutual Information

Peak information valuesbi

ts

Observation of artificial movement

Relative timing between neural modulationand cursor velocity

Tuning to Cursor Direction: Cosine tuning curves

Difference between preferred directions in action & action observation conditions

What role does the target and cursor play?

Target & Cursor Target only Cursor only

• Visible target• Visible cursor

• Visible target• Invisible cursor

• Invisible target• Visible cursor

y

x

Action Observation Phase

Monkey voluntarily maintains a still posture while observing video playback

Results: Peri-target Hit Spike Rate Histograms

Cells modulate in a very similar way during action and action observation

Results: Mutual Information

Neural activity conveys information about cursor velocity during action observation

MIMI MI MI

Lag: Lag of movement with respect to neural activity

Positive Lag: Movement follows neural activity

Negative Lag: Movement precedes neural activity

Possible explanations:

Our Conclusion:

Neural activity is due to the covert generation of a motor plan

BrainGateTM Pilot Device

Sensor

Cable

Cart

BrainGateTM Sensor Implantation and Post-Op Recovery as Planned

• Surgery as planned

• Post-op recovery unremarkable

• Wound healing around pedestal complete

Arrayon Cortex

Insertion

2 months post implant

Binary Modulation-Imagined Opening/Closing of Hand

• Can we recover the link from the body back to the brain by creating proprioception?

Answer: Yes, by using the arm as a proprioceptive channel

Real-time Experimental Set-up

PD controller attempts to match robot position withcursor position

Real-time Decoding using Singularity Robust Pseudo-Inverse

( ) 1−= TT XXXYW( )∑ −=

pjjpjp yyE

,

2ˆError Function: Solution:

( ) ∑∑ +−=j

jpj

jpjp wyyE 2

,

2ˆ λ

Minimize prediction error while also keeping the model flat (or conservative)

Andy Fagg

Real-time Decoding Conditions

• Arm control (Arm)Monkey performs task with its arm

• Brain control – Visual feedback only (Brain-Vision)Monkey is trained to keep its arm motionless

• Brain control – Dual Feedback (Brain-Dual)Monkey’s arm is moved to follow the cursor

Cursor position Arm position

Real-time controlperformance

Real-time Decoding Conditions

• Arm control (Arm)Monkey performs task with its arm

• Brain control – Visual feedback only (Brain-Vision)Monkey is trained to keep its arm motionless

• Brain control – Dual Feedback (Brain-Dual)Monkey’s arm is moved to follow the cursor

• Brain control – Dual Feedback (Brain-Dual Noisy)Monkey’s arm is moved to follow a noisy trajectory that doesnot match the cursor’s trajectory

Why may proprioceptive feedback improve cursor control?

Examples Of MI Profiles and MI Peak Shifts:

• Range of shading around each MI lag is –/+ 1 STD

• Statistical significance of MI at each lag is calculated using paired-sample Wilcoxon signed rank test (p<0.001):-Test sample: MI values at each -12 to +12 lags centered on each lag, from all bootstrap iterations at those lags

Kinarm ConditionVisual Fdbck ConditionDual Fdbck Condition

Summary of MI Peak Shifts:

Shown cells had statistically significant MI peak values in each condition

Distribution of Peak MI Lags for Three Experimental Conditions

Kinarm Condition Visual Fdbck Condition Dual Fdbck Condition

Hatsopoulos LabAdam DickeySunday Francis Julian MattielloDawn PaulsenBen PerroneJake ReimerMaryam SalehAaron SuminskiTaka TakahashiDennis Tkach

Andrew Fagg, Univ. of Oklahoma

Funding

NIH R01 NS45853NIH R01 NS048845