neuromorphic vlsi event-based devices and systems/file/giacomo_indiveri.pdf · 2012-06-27 ·...

Neuromorphic VLSI Event-Based devices and systems

Giacomo Indiveri

Institute of NeuroinformaticsUniversity of Zurich and ETH Zurich

LTU, LuleaMay 28, 2012

G.Indiveri (http://ncs.ethz.ch/) Neuromorphic spiking chips 1 / 38

Outline

1 “Neuromorphic Engineering”

2 Spike-based sensory systems

3 Spiking Neural NetworksSilicon NeuronsSilicon synapsesWinner-Take-All networksMulti-chip networks

4 Learning

5 Neuromorphic Cognitive Systems


Natural ComputationThe Honeybee

Energy consumption: 10−15 J/op, at least 106

more efficient than digital silicon(20watts vs. 1Mil.watts)

The brain of the workerhoneybee occupies a volumeof around 1mm3 and weighsabout 1mg.The total number of neurons inthe brain is estimated to be950,000

Flies acrobatically

Recognizes patterns

Navigates

Forages

Communicates


Neocortex → Neural computation → Silicon→ Behavior

Synaptic Inputs Constant current Synapse

Soma

0 0.05 0.1 0.15 0.20

0.2

0.4

0.6

0.8

1

Time (s)

Vm

em

(V)


Neuromorphic VLSI systems

100µ

[Nuno da Costa, INI, 2008]

Goals:

To exploit the physics of silicon to reproduce the bio-physics of neuralsystems, using subthreshold analog VLSI circuits.

To develop multi-chip spike-based computing systems, using theAddress-Event Representation (AER) and asynchronous digital VLSItechnology.

To automatically configure and “program” neuromorphic processingsystems distributed across multiple chips, to carry out real–timebehavioral tasks.


Outline




4 Learning



AER silicon retinasTobi Delbruck


Silicon retina propertieshttp://siliconretina.ini.uzh.ch

(movie)G.Indiveri (http://ncs.ethz.ch/) Neuromorphic spiking chips 8 / 38

An AER silicon cochleaShih-Chii Liu


Silicon cochlea properties


Outline




4 Learning



Implement neural computation in silicon

Classical neural networks

w1

w2

w3

w4

wn

Post-synaptic Output

Pre

-syn

aptic

Inpu

ts

Neuromorphic multi-neuron networks

w2 w3 w4 wnw1

Pre-synaptic Inputs



Spiking multi-neuron architectures

Networks of silicon neurons with adaptation,refractory period, etc.

Silicon synapses with realistic temporal dynamics

Winner-Take-All architectures

Spike-based plasticity mechanisms


Outline




4 Learning



Silicon neuronsThe low-power adaptive exponential I&F neuron

Cahp

Cmem

Vtaua

Vahp

Vspk

Vtau

Vmem

Vrf

Positive Feedback

Refractory Period

Iin

Leak

Adaptation

Vthr

M1

M3 M2

VrestVthra

M8 M7

M4

M5

M9

M6

M10

M11

M12

M13

M14

M15

M17

M16

M19

M21

M20

M18

M22

Imem

Iahp

Ifb

DPI

DPI

τddt

Imem + Imem ≈Ig IinIτ

+ f (Imem)

[Indiveri et al., ISCAS 2010]


http://www.ini.uzh.ch/~giacomo/papers/pdf/iscas10c.pdf

The low-power I&F neuronPositive Feedback

0 5 10 15 20 25 30 351

1.2

1.4

1.6

1.8

2

2.2

2.4

2.6

2.8

Time (ms)

I mem

/I 0

data

fit


The low-power I&F neuronSpike frequency adaptation

0 5 10 15 202

4

6

8

10

12

14

Spike count

Inst

anta

neou

s fir

ing

rate

(H

z)

0 1 2 3 4 50

1

2

3

4


The low-power I&F neuronBasic response properties

Leaky I&F model

0 0.05 0.1 0.15 0.20

0.2

0.4

0.6

0.8

1

Time (s)

Mem

bran

e po

tent

ial (

V)

F-F curve (note mismatch)


Outline




4 Learning



Synapses

Real synapses Artificial synapses

w1

w2

w3

w4

wn


Pre

-syn

aptic

Inpu

ts

Synapses are often modeled asinstantaneous multipliers.

Science and Engineering Visualization Challenge

2005 winner, Graham Johnson, Medical Media, Boulder,

Colorado.


The diff-pair integrator (DPI) circuit

Vw

Vthr

Mτ

Min Mthr

Mw

Vτ

Msyn

Isyn

Vsyn

Csyn

Mpre

Iw

Iin

Iτ

Isyn(t) = I0e−κ

UT(Vsyn(t)−Vdd )

Ithr = I0e−κ(Vthr−Vdd )

UT

Csynddt

Vsyn =−(Iin− Iτ)

τddt

Isyn + Isyn =Ithr Iw

Iτ

[Bartolozzi and Indiveri, Neural Computation, 2007]


http://www.ini.uzh.ch/~giacomo/papers/pdf/Bartolozzi_Indiveri07.pdf

The DPI synapseTemporal dynamics

0 0.5 1 1.5 2 2.5 3 3.50

50

100

150

200

250

300

350

400

450

Time (s)

EP

SC

(nA

)

Vw

=300mV

Vw

=320mV

Vw

=340mV


Outline




4 Learning



Recurrent cooperative-competitive architectures

Hardwired localsynapses

Local excitatoryconnections

Global inhibitoryconnections

[Chicca et al., Nips, 2006]


Local recurrent connectivityWinner-take-All architectures

0 2 4 6 8 100

5

10

15

20

25

30

Time (s)

Neu

ron

addr

ess

Input Stimulus

0 50 100Mean f (Hz)

AER INPUT Y

AER

INPU

T X

AER O

UTPU

T

Input signals are encodedwith mean firing rates

Computation andinformation transfer isdata driven



0 2 4 6 8 100

5

10

15

20

25

30

Time (s)

Neu

ron

addr

ess

Feedforward Network

0 20 40Mean f (Hz)

Without local connectivityactivated output spike ratesrepresent linearly input spikerates (modulo mismatcheffects)



0 2 4 6 8 100

5

10

15

20

25

30

Time (s)

Neu

ron

addr

ess

Feedback Network

0 20 40Mean f (Hz)

With local WTA connectivity thenetwork exhibits:

Selective amplification

Signal normalization

Signal restoration

[Chicca et al., Nips, 2006]


Outline




4 Learning



Spikes and Address-Event Systems

1 2 3 2 3 12

Inputs

Encode Decode

Address Event Bus

SourceChip

Outputs

DestinationChip

Action Potential

Address-Eventrepresentation ofaction potential

21

321

3

0 0.05 0.1 0.15 0.20

0.2

0.4

0.6

0.8

1

Time (s)

Vm

em (

V)


Hierarchical or multi-layer networks

The basic problem with these models is, of course, generalization:a look-up table cannot deal with new events, such as viewing a facefrom the side rather than the front, and it cannot learn in the predic-tive sense described earlier. One of the simplest and most powerfultypes of algorithm developed within learning theory corresponds tonetworks that combine the activities of ‘units’, each broadly tuned toone of the examples (Box 1). Theory (see references in Box 1) showsthat a combination of broadly tuned neurons — those that respondto a variety of stimuli, although at sub-maximal firing rates — mightgeneralize well by interpolating among the examples.

In visual cortex, neurons with a bell-shaped tuning are common.Circuits in infratemporal cortex and prefrontal cortex, which com-bine activities of neurons in infratemporal cortex tuned to differentobjects (and object parts) with weights learned from experience, mayunderlie several recognition tasks, including identification andcategorization. Computer models have shown the plausibility of thisscheme for visual recognition and its quantitative consistency withmany data from physiology and psychophysics2–5 .

Figure 2 sketches one such quantitative model, and summarizes aset of basic facts about cortical mechanisms of recognition establishedover the last decade by several physiological studies of cortex6–8. Objectrecognition in cortex is thought to be mediated by the ventral visualpathway running from primary visual cortex, V1, over extrastriatevisual areas V2 and V4 to the inferotemporal cortex. Starting fromsimple cells in V1, with small receptive fields that respond preferably tooriented bars, neurons along the ventral stream show an increase inreceptive field size as well as in the complexity of their preferred stimuli.At the top of the ventral stream, in the anterior inferotemporal cortex,neurons respond optimally to complex stimuli such as faces and otherobjects. The tuning of the neurons in anterior inferotemporal cortexprobably depends on visual experience9–19. In addition, some neuronsshow specificity for a certain object view or lighting condition13,18,20–22.For example, Logothetis et al.13 trained monkeys to perform an objectrecognition task with isolated views of novel three-dimensional objects(‘paperclips’; Fig. 1). When recording from the animals' inferotemporalcortex, they found that the great majority of neurons selectively tunedto the training objects were view-tuned (see Fig. 1) to one of the trainingobjects. About one tenth of the tuned neurons were view-invariant,consistent with an earlier computational hypothesis23.

In summary, the accumulated evidence points to a visual recog-nition system in which: (1) the tuning of infratemporal cortex cells isobtained through a hierarchy of cortical stages that successivelycombines responses from neurons tuned to simpler features; and (2)the basic ability to generalize depends on the combination of cellstuned by visual experience. Notice that in the model of Fig. 2, thetuning of the units depends on learning, probably unsupervised (forwhich several models have been suggested24; see also review in thisissue by Abbott and Regehr, page 796), since it depends only onpassive experience of the visual inputs. However, the weights of thecombination (see Fig. 3) depend on learning the task and require atleast some feedback (see Box 2).

Thus, generalization in the brain can emerge from the linear com-bination of neurons tuned to an optimal stimulus — effectivelydefined by multiple dimensions25,23,26. This is a powerful extension ofthe older computation-through-memory models of vision andmotor control. The question now is whether the available evidencesupports the existence of a similar architecture underlying general-ization in domains other than vision.

insight review articles

Figure 1 Tuned units in inferotemporal cortex. A monkey was trained to recognizea three-dimensional ‘paperclip’ from all viewpoints (pictured at top). The graphshows tuning to the multiple parameters characterizing each view summarized interms of spike rate versus rotation angle of three neurons in anterior inferotemporalcortex that are view-tuned for the specific paperclip. (The unit corresponding to thegreen tuning curve has two peaks — to a view of the object and its mirror view.) Acombination of such view-tuned neurons (Fig. 2) can provide view-invariant, objectspecific tuning as found in a small fraction of the recorded neurons. Adapted fromLogothetis et al.13.

–80 –30 –0 60 100

Sp

ikes

per

sec

ond

25

20

15

10

0

5

–180 –120 –60 0 60 120 180

213

216

239

Cells

Rotation around y axis (degrees)

Figure 2 A model of visual learning. The model summarizes in quantitative termsother models and many data about visual recognition in the ventral stream pathwayin cortex. The correspondence between the layers in the model and visual areas isan oversimplification. Circles represent neurons and arrows represent connectionsbetween them; the dots signify other neurons of the same type. Stages of neuronswith bell-shaped tuning (with black arrow inputs), that provide example-basedlearning and generalization, are interleaved with stages that perform a max-likeoperation3 (denoted by red dashed arrows), which provides invariance to positionand scale. An experimental example of the tuning postulated for the cells in thelayer labelled inferotemporal in the model is shown in Fig. 1. The model accountswell for the quantitative data measured in view-tuned inferotemporal cortex cells10

(J. Pauls, personal communication) and for other experiments55. Superposition ofgaussian-like units provides generalization to three-dimensional rotations andtogether with the soft-max stages some invariance to scale and position. IT,infratemporal cortex, AIT, anterior IT; PIT, posterior IT; PFC, prefrontal cortex.Adapted from M. Riesenhuber, personal communication.

V1

V4

V1

PFC

AIT

IT

Categ. Ident.

V4/PIT

NATURE | VOL 431 | 14 OCTOBER 2004 | www.nature.com/nature 769

14.10 Insight 768 Poggio 1/10/04 7:48 pm Page 769

© 2004 Nature Publishing Group

AER INPUT Y

AER

INPU

T X

AER O

UTPU

T


Outline




4 Learning



Spike-timing dependent plasticity (STDP)

Abbot, Nelson, 2000

1 If an input (pre-synaptic) spike arrives shortlybefore an output (post-synaptic) spike is emitted,the synaptic efficacy is increased.

2 If it arrives soon after the output spike is emitted,the synaptic efficacy is decreased.


STDP is not enoughfor learning complex spatio-temporal patterns

Senn, Biological Cybernetics, 2002[...] additional non linearities are required ifSTDP should be relevant for both encodinginformation represented in a spike correlationcode and a mean rate code without spikecorrelations.


STDP is not enoughfor learning complex spatio-temporal patterns

Spike-based learning mechanisms idealfor VLSI implementations

depend on the neuron’s membranepotential;

synaptic weights have two stable states(bi-stability);

many synapses see the same pre- andpost-synaptic mean activity (redundancy);

LTP/LTD is induced only in a randomsubset of stimulated synapses(stochasticity).

[Fusi et al. 2000]; [Gütig, Sompolinsky 2006]; [Brader et al. 2007]


Spike-driven plasticity in silicon

w2 w3 w4 wnw1

Pre-synaptic Inputs


Pre-synaptic component

Diff-pairIntergator

AER inputspike

V’UP

Vwth

V’DN

VWi

Vmem

Vwhi

Vwlow

Vilk

pre

~pre

Isyn

Post-synaptic component

Comparator

I&F CircuitDiff-pair

IntegratorVCa

Vcmp

Vmth

CurrentComparator

VUP

VDN

Vmem

Vmem

Vspk


Spike-driven plasticity in silicon

Dn

θ

Up

~w

eigh

tpo

st

0 0.1 0.2 0.3 0.4

pre

Time(s)

[Mitra et al. 2009]


Stochastic weight updateLTP/LTD probabilities and stop-learning

LTD consolidation

VL

θ

VH

wV

mem

0 0.1 0.2 0.3 0.4

pre

Time(s)

No LTD consolidation

VL

θ

VH

wV

mem

0 0.1 0.2 0.3 0.4

pre

Time(s)


Outline




4 Learning



Distributed event-driven systems

Neuromorphic “cognitive” systems can be assembled by using:1 Full custom hybrid analog/digital neural processing VLSI devices.2 A spike based communication protocol (e.g., the Address-Event

Representation).3 Systematic methods for parameter tuning.4 Methods for implementing state-dependent computation using spiking

neural networks.


State dependent computation

sWTA diagramExcitatory

neurons

Global

Inhibition

Inhibitory

neurons }Nearest-N

Excitation

I E E E E E

sWTA networks as building blocks

Linearbehaviors

Non linearbehaviors

Analog gain Locus invariance Gain control bycommon mode input

Selective amplification Signal restoration Multi-stability

[Douglas and Martin, 2007]

Configure key parameters of the WTA network automatically.Implement “state-holding” elements.Learn network connectivity patterns.


ConclusionsToward neuromorphic cognitive behaving systems

By using event based sensors andspike-based neural processingcircuits it is possible to implementreal-time sensory-motor systems.

By using soft WTA multi-chipnetworks it is possible toimplement real-timestate-dependent computation.

The AER communicationinfrastructure and automatedparameter tuning techniques,allow us to synthesizespike-based neural finite statemachines.


Acknowledgments

The Institute of NeuroinformaticsElisabetta ChiccaStefano FusiChiara BartolozziThe NCS group (http://ncs.ethz.ch/)

Rodney DouglasKevan MartinRichard Hahnloser

Funding sourcesneuroP (257219) ERCSCANDLE (ICT-231168)eMorph (ICT-231467)

nAttention (121713) SNFSoundRec (119973) SNF


http://ncs.ethz.ch/

neuromorphic vlsi event-based devices and systems/file/giacomo_indiveri.pdf · 2012-06-27 ·...

Documents