a model of invariant object recognition in the visual ...milward00.pdf · invariant object...

LETTER Communicated by Laurenz Wiskott

A Model of Invariant Object Recognition in the VisualSystem Learning Rules Activation Functions LateralInhibition and Information-Based Performance Measures

Edmund T RollsT MilwardOxford University Department of Experimental Psychology Oxford OX1 3UDEngland

VisNet2 is a model to investigate some aspects of invariant visual objectrecognition in the primate visual system It is a four-layer feedforwardnetwork with convergence to each part of a layer from a small region ofthe preceding layer with competition between the neurons within a layerand with a trace learning rule to help it learn transform invariance Thetrace rule is a modied Hebbian rule which modies synaptic weights ac-cording to both the current ring rates and the ring rates to recently seenstimuli This enables neurons to learn to respond similarly to the gradu-ally transforming inputs it receives which over the short term are likelyto be about the same object given the statistics of normal visual inputsFirst we introduce for VisNet2 both single-neuron and multiple-neuroninformation-theoretic measures of its ability to respond to transformedstimuli Second using these measures we show that quantitatively re-setting the trace between stimuli is not necessary for good performanceThird it is shown that the sigmoid activation functions used in VisNet2which allow the sparseness of the representation to be controlled allowgood performance when using sparse distributed representations Fourthit is shown that VisNet2 operates well with medium-range lateral inhibi-tion with a radius in the same order of size as the region of the precedinglayer from which neurons receive inputs Fifth in an investigation of dif-ferent learning rules for learning transform invariance it is shown thatVisNet2 operates better with a trace rule that incorporates in the traceonly activity from the preceding presentations of a given stimulus withno contribution to the trace from the current presentation and that thisis related to temporal difference learning

1 Introduction

11 Background There is evidence that over a series of cortical pro-cessing stages the visual system of primates produces a representation ofobjects that shows invariance with respect to for example translation sizeand view as shown by recordings from single neurons in the temporal lobe

Neural Computation 12 2547ndash2572 (2000) cdeg 2000 Massachusetts Institute of Technology

2548 Edmund T Rolls and T Milward

(Desimone 1991 Rolls 1992 Rolls amp Tovee 1995 Tanaka Saito Fukadaamp Moriya 1991) (see Figure 2) Rolls (1992 1994 1995 1997 2000) has re-viewed much of this neurophysiological work and has advanced a theoryfor how these neurons could acquire their transform-independent selectiv-ity based on the known physiology of the visual cortex and self-organizingprinciples (see also Wallis amp Rolls 1997 Rolls amp Treves 1998)

Rollsrsquos hypothesis has the following fundamental elements

A series of competitive networks organized in hierarchical layers ex-hibiting mutual inhibition over a short range within each layer Thesenetworks allow combinations of features or inputs that occur in a givenspatial arrangement to be learned by neurons ensuring that higher-order spatial properties of the input stimuli are represented in thenetwork

A convergent series of connections from a localized population of cellsin preceding layers to each cell of the following layer thus allowing thereceptive-eld size of cells to increase through the visual processingareas or layers

A modied Hebb-like learning rule incorporating a temporal trace ofeach cellrsquos previous activity which it is suggested will enable theneurons to learn transform invariances (see also Foldiak 1991)

Based on these hypotheses Wallis and Rolls produced a model of ventralstream cortical visual processing designed to investigate computational as-pects of this processing (Wallis Rolls amp Foldiak 1993 Wallis amp Rolls 1997)With this model it has been shown that provided that a trace learning ruleto be dened below is used the network (VisNet) can learn transform-invariant representations It can produce neurons that respond to some butnot other stimuli with translation and view invariance In investigations oftranslation invariance described by Wallis and Rolls (1997) the stimuli weresimple stimuli such as T L and C or more complex stimuli such as facesRelatively few (up to seven) different stimuli such as different faces wereused for training

The work described here investigates several key issues that inuencehow the network operates including new formulations of the trace rule thatsignicantly improve the performance of the network how local inhibitionwithin a layer may improve the performance of the network how neuronswith a sigmoid activation function that allows the sparseness of the repre-sentation to be controlled can improve the performance of the network andhow the performance of the network scales up when a larger number oftraining locations and a larger number of stimuli are used The model de-scribed previously by Wallis and Rolls (1997) was modied in the ways justdescribed to produce the model described here which is denoted VisNet2

Before describing these new results we rst outline the architecture ofthe model In section 2 we describe a way to measure the performance of

Invariant Object Recognition in the Visual System 2549

the network using information theory measures This approach has a dou-ble advantage it is based on information theory which is an appropriatemeasure for how any information processing system performs and it usesthe same measurement that has been applied recently to measure the per-formance of real neurons in the brain (Rolls Treves Tovee amp Panzeri 1997Rolls Treves amp Tovee 1997 Booth amp Rolls 1998) and thus allows directcomparisons in the same units bits of information to be made between thedata from real neurons and those from VisNet We then introduce the newform of the trace rule In section 3 we show using the new informationtheory measure how the new version of the trace rule operates in VisNetand how the parameters of the lateral inhibition inuence the performanceof the network and we describe how VisNet operates when using largernumbers of stimuli than were used previously together with values for thesparseness of the ring that are closer to those found in the brain (Wallis ampRolls 1997 Rolls amp Treves 1998)

12 The Trace Rule The learning rule implemented in the simulationsuses the spatiotemporal constraints placed on the behavior of real-worldobjects to learn about natural object transformations By presenting consis-tent sequences of transforming objects the cells in the network can learn torespond to the same object through all of its naturally transformed statesas described by Foldiak (1991) and Rolls (1992) The learning rule incorpo-rates a decaying trace of previous cell activity and is henceforth referredto simply as the trace learning rule The learning paradigm is intended inprinciple to enable learning of any of the transforms tolerated by inferiortemporal cortex neurons (Rolls 1992 1995 1997 2000 Rolls amp Tovee 1995Rolls amp Treves 1998 Wallis amp Rolls 1997)

The trace update rule used in the simulations by Wallis and Rolls (1997) isequivalent to that of Foldiak and the earlier rule of Sutton and Barto (1981)and can be summarized as follows

Dwj D ayt xj

where

yt D (1 iexcl g)yt C gyt iexcl1

xj j th input to the neuron

yt Trace value of the output of the neuron at time step t

wj Synaptic weight between j th input and the neuron

y Output from the neuron

a Learning rate annealed between unity and zero

g Trace value the optimal value varies with presentation sequencelength


Rec

eptiv

e Fi

eld

Siz

e d

eg

Layer 4

Layer 3

Layer 2

Layer 1

Figure 1 Stylized image of the VisNet four-layer network Convergencethrough the network is designed to provide fourth-layer neurons with infor-mation from across the entire input retina

To bound the growth of each cellrsquos dendritic weight vector its length isexplicitly normalized (Wallis amp Rolls 1997) as is common in competitivenetworks (see Rolls amp Treves 1998)

13 The Network Figure 1 shows the general convergent network ar-chitecture used The architecture of VisNet 2 is similar to that of VisNet ex-cept that neurons with sigmoid activation functions are used with explicitcontrol of sparseness the radius of lateral inhibition and thus of compe-tition is larger as specied below different versions of the trace learningrule as described below are used and an information-theoretic measure ofperformance replaces that used earlier by Wallis and Rolls (1997) The net-work itself is designed as a series of hierarchical convergent competitivenetworks with four layers specied as shown in Table 1 The forward con-nections to a cell in one layer come from a small region of the precedinglayer dened by the radius in Table 1 which will contain approximately67 of the connections from the preceding layer

VisNet is provided with a set of input lters that can be applied to animage to produce inputs to the network that correspond to those providedby simple cells in visual cortical area 1 (V1) The purpose is to enable withinVisNet the more complicated response properties of cells between V1 and


Table 1 VisNet Dimensions

Dimensions Number of Connections Radius

Layer 4 32 pound 32 100 12Layer 3 32 pound 32 100 9Layer 2 32 pound 32 100 6Layer 1 32 pound 32 272 6Retina 128 pound 128 pound 32 mdash mdash

50

0 3213 80 20 50

20

TE

TEO

V4

V2

V1

LGN

combinations of features

larger receptive fields

Rec

epti

ve F

ield

Siz

e d

eg

Eccentricity deg

configuration sensitiveview dependent

view independence

80

32

13

Figure 2 Convergence in the visual system V1 visual cortex area V1 TEOposterior inferior temporal cortex TE inferior temporal cortex (IT) (Adaptedfrom Rolls 1992)

the inferior temporal cortex (IT) to be investigated using as inputs naturalstimuli such as those that could be applied to the retina of the real visualsystem This is to facilitate comparisons between the activity of neurons inVisNet and those in the real visual system to the same stimuli The elon-gated orientation-tuned input lters used accord with the general tuningproles of simple cells in V1 (Hawken amp Parker 1987) and are computed byweighting the difference of two gaussians by a third orthogonal gaussian asdescribed by Wallis et al (1993) and Wallis and Rolls (1997) Each individuallter is tuned to spatial frequency (00625 to 05 cycles pixelsiexcl1 over fouroctaves) orientation (0ndash135 degrees in steps of 45 degrees) and sign (C

iexcl1)Of the 272 layer 1 connections of each cell the number from each group isas shown in Table 2

Graded local competition is implemented in VisNet in principle by alateral inhibition-like scheme The two steps in VisNet were as followsA local spatial lter was applied to the neuronal responses to implement


Table 2 VisNet Layer 1 Connectivity

Frequency 00625 0125 025 05

Number of connections 201 50 13 8

Note The frequency is in cycles per pixel

lateral inhibition Contrast enhancement of the ring rates (as opposed towinner-take-all competition) was realized (Wallis and Rolls 1997) by rais-ing the ring rates r to a xed power p and then renormalizing the rates(ie y D rp (

Pi rp

i ) where y is the ring rate after the competition) Thebiological rationale for this is the greater than linear increase in neuronalring as a function of the activation of the neuron This characterizes therst steeply rising portion close to threshold of the sigmoid-like activationfunction of real neurons In this article we introduce for VisNet2 a sigmoidactivation function The general calculation of the response of a neuron isas described by Wallis and Rolls (1997) and Wallis et al (1993) In briefthe postsynaptic ring is calculated as shown in equation 21 where in thisarticle the function f incorporates a sigmoid activation function and lateralinhibition as described in section 2

The measure of network performance used in VisNet the Fisher met-ric reects how well a neuron discriminates between stimuli compared tohow well it discriminates between different locations (or more generally theimages used rather than the objects each of which is represented by a setof images over which invariant stimulus or object representations must belearned) The Fisher measure is very similar to taking the ratio of the two Fvalues in a two-way ANOVA where one factor is the stimulus shown andthe other factor is the position in which a stimulus is shown The measuretakes a value greater than 10 if a neuron has more different responses to thestimuli than to the locations Further details of how the measure is calcu-lated are given by Wallis and Rolls (1997) Measures of network performancebased on information theory and similar to those used in the analysis of thering of real neurons in the brain (see Rolls and Treves 1998) are introducedin this article for VisNet2 They are described in section 2 compared to theFisher measure at the start of section 3 and then used for the remainder ofthe article

2 Methods

21 Modied Trace Learning Rules The trace rule that has been usedin VisNet can be expressed by equation 24 where t indexes the currenttrial and t iexcl1 the previous trial (The superscript on w indicates the versionof the learning rule) This is similar to Sutton and Barto (1981) and one ofthe formulations of Foldiak (1991) The postsynaptic term in the synaptic


modication is based on the trace from previous trials available at timet iexcl 1 and on the current activity (at time t ) with g determining the rela-tive proportion of these two This is expressed in equation 23 where y isthe postsynaptic ring rate calculated from the presynaptic ring rates xj assummarized in equation 21 and yt is the postsynaptic trace at time t In thisarticle we compare this type of rule with a different type in which the post-synaptic term depends on only the trace left from previous trials availablefrom t iexcl1 without including a contribution from the current instantaneousactivity y of the neuron This is expressed in equation 25 In addition we in-troduce a comparison of a trace incorporated in the presynaptic term (withthe two variants for times t and t iexcl 1 shown in equations 26 and 27)The presynaptic trace xt

j is calculated as shown in equation 22 We alsointroduce a comparison of traces incorporated in both the presynaptic andthe postsynaptic terms (with the two variants for times t and t iexcl 1 shownin equations 28 and 29) We will demonstrate that using the trace at timet iexcl 1 is signicantly better than the trace at time t but that the type oftrace (presynaptic postsynaptic or both) does not affect performance Whatis achieved computationally by these different rules and their biologicalplausibility are considered in section 4

Following are these equations

y D fplusmnX

wjxj

sup2(21)

xtj D (1 iexcl g)xt

j C gxt iexcl1j (22)

yt D (1 iexcl g)yt C gyt iexcl1 (23)

Dw1j yt xt

j (24)

Dw2j yt iexcl1xt

j (25)

Dw3j yt xt

j (26)

Dw4j yt xt iexcl1

j (27)

Dw5j yt xt

j C yt xtj D Dw1

j C Dw3j (28)

Dw6j yt iexcl1xt

j C yt xt iexcl1j D Dw2

j C Dw4j (29)

22 Measures of Network Performance Based on Information TheoryTwo new performance measures based on information theory are intro-duced The information-theoretic measures have the advantages that theyprovide a quantitative and principled way of measuring the performanceof an information processing system and are essentially identical to thosethat are being applied to analyze the responses of real neurons (see Rolls ampTreves 1998) thus allowing direct comparisons of real and model systemsThe rst measure is applied to a single cell of the output layer and mea-sures how much information is available from a single response of the cell


to each stimulus s Each response of a cell in VisNet to a particular stimulusis produced by the different transforms of the stimulus For example ina translation-invariance experiment each stimulus image or object mightbe shown in each of nine locations so that for each stimulus there wouldbe nine responses of a cell The set of responses R over all stimuli wouldthus consist of 9NS responses where NS is the number of stimuli If there islittle variation in the responses to a given stimulus and the responses to thedifferent stimuli are different then considerable information about whichstimulus was shown is obtained on a single trial from a single response Ifthe responses to all the stimuli overlap then little information is obtainedfrom a single response about which stimulus (or object) was shown Thusin general the more information about a stimulus that is obtained the bet-ter is the invariant representation The information about each stimulus iscalculated by the following formula with the details of the procedure givenby Rolls Treves Tovee and Panzeri (1997) and background to the methodgiven there and by Rolls and Treves (1998)

I(s R) DX

r2R

P(r | s) log2P(r | s)

P(r) (210)

The stimulus-specic information I(s R) is the amount of informationthat the set of responses R has about a specic stimulus s The mutualinformation between the whole set of stimuli S and of responses R is the av-erage across stimuli of this stimulus-specic information (see equation 211)(Note that r is an individual response from the set of responses R)

The calculation procedurewas identical to that described by Rolls TrevesTovee and Panzeri (1997) with the following exceptions First no correctionwas made for the limited number of trials because in VisNet2 (as in VisNet)each measurement of a response is exact with no variation due to samplingon different trials Second the binning procedure was altered in such a waythat the ring rates were binned into equispaced rather than equipopulatedbins This small modication was useful because the data provided by Vis-Net2 can produce perfectly discriminating responses with little trial-to-trialvariability Because the cells in VisNet2 can have bimodally distributed re-sponses equipopulated bins could fail to separate the two modes perfectly(This is because one of the equipopulated bins might contain responses fromboth of the modes) The number of bins used was equal to or less than thenumber of trials per stimulus that is for VisNet the number of positions onthe retina (Rolls Treves Tovee amp Panzeri 1997)

Because VisNet operates as a form of competitive net to perform cate-gorization of the inputs received good performance of a neuron will becharacterized by large responses to one or a few stimuli regardless of theirposition on the retina (or other transform) and small responses to the otherstimuli We are thus interested in the maximum amount of information thata neuron provides about any of the stimuli rather than the average amount


of information it conveys about the whole set S of stimuli (known as themutual information) Thus for each cell the performance measure was themaximum amount of information a cell conveyed about any one stimulus(with a check in practice always satised that the cell had a large responseto that stimulus as a large response is what a correctly operating competi-tive net should produce to an identied category) In many of the graphs inthis article the amount of information that each of the 100 most informativecells had about any stimulus is shown

If all the output cells of VisNet learned to respond to the same stimulusthen the information about the set of stimuli S would be very poor andwould not reach its maximal value of log2 of the number of stimuli (in bits)A measure that is useful here is the information provided by a set of cellsabout the stimulus set If the cells provide different information becausethey have become tuned to different stimuli or subsets of stimuli then theamount of this multiple cell information should increase with the number ofdifferent cells used up to the total amount of information needed to specifywhich of the NS stimuli have been shown that is log2 NS bits Procedures forcalculating the multiple cell information have been developed for multipleneuron data by Rolls Treves and Tovee (1997) (see also Rolls amp Treves1998) and the same procedures were used for the responses of VisNet Inbrief what was calculated was the mutual information I (S R) that is theaverage amount of information that is obtained from a single presentation ofa stimulus from the responses of all the cells For multiple cell analysis theset of responses R consists of response vectors comprising the responsesfrom each cell Ideally we would like to calculate

I(S R) DX

s2S

P(s)I(s R) (211)

However the information cannot be measured directly from the prob-ability table P(r s) embodying the relationship between a stimulus s andthe response rate vector r provided by the ring of the set of neurons to apresentation of that stimulus (Note as is made clear at the start of this arti-cle that stimulus refers to an individual object that can occur with differenttransformsmdashfor example as translation here but elsewhere view and sizetransforms See Wallis and Rolls 1997) This is because the dimensional-ity of the response vectors is too large to be adequately sampled by trialsTherefore a decoding procedure is used in which the stimulus s0 that gaverise to the particular ring-rate response vector on each trial is estimatedThis involves for example maximum likelihood estimation or dot productdecoding (For example given a response vector r to a single presentationof a stimulus its similarity to the average response vector of each neuronto each stimulus is used to estimate using a dot product comparison whichstimulus was shown The probabilities of it being each of the stimuli canbe estimated in this way Details are provided by Rolls Treves and Tovee


(1997) and by Panzeri Treves Schultz and Rolls (1999)) A probability tableis then constructed of the real stimuli s and the decoded stimuli s0 From thisprobability table the mutual information between the set of actual stimuliS and the decoded estimates S0 is calculated as

I(S S0 ) DX

ss0

P(s s0 ) log2P(s s0 )

P(s)P(s0 ) (212)

This was calculated for the subset of cells that had as single cells themost information about which stimulus was shown Often ve cells foreach stimulus with high information values for that stimulus were used forthis

23 Lateral Inhibition Competition and the Neuronal ActivationFunction As originally conceived (Wallis amp Rolls 1997) the essentiallycompetitive networks in VisNet had local lateral inhibition and a steeplyrising activation function which after renormalization resulted in the neu-rons with the higher activations having very much higher ring rates rel-ative to other neurons The idea behind the lateral inhibition apart fromthis being a property of cortical architecture in the brain was to prevent toomany neurons receiving inputs from a similar part of the preceding layer re-sponding to the same activity patterns The purpose of the lateral inhibitionwas to ensure that different receiving neurons coded for different inputsThis is important in reducing redundancy (Rolls amp Treves 1998) The lateralinhibition was conceived as operating within a radius that was similar tothat of the region within which a neuron received converging inputs fromthe preceding layer (because activity in one zone of topologically organizedprocessing within a layer should not inhibit processing in another zone inthe same layer concerned perhaps with another part of the image) How-ever the extent of the lateral inhibition actually investigated by Wallis andRolls (1997) in VisNet operated over adjacent pixels Here we investigate inVisNet2 lateral inhibition implemented in a similar but not identical waybut operating over a larger region set within a layer to approximately halfof the radius of convergence from the preceding layer

The lateral inhibition and contrast enhancement just described is actuallyimplemented in VisNet2 in two stages Lateral inhibition is implemented inthe experiments described here by convolving the activation of the neuronsin a layer with a spatial lter I where d controls the contrast and s controlsthe width and a and b index the distance away from the center of the lter

Iab D

8gtltgt

iexcldeiexcl a2 C b2

s2 if a 6D 0 or b 6D 0

1 iexclP

a 6D0b 6D0

Iab if a D 0 and b D 0

This is a lter that leaves the average activity unchanged


Table 3 Sigmoid Parameters for the 25 Location Runs

Layer 1 2 3 4

Percentile 992 98 88 91

Slope b 190 40 75 26

The second stage involves contrast enhancement In VisNet this wasimplemented by raising the activations to a xed power and normalizingthe resulting ring within a layer to have an average ring rate equal to 10Here in VisNet2 we compare this to a more biologically plausible form ofthe activation function a sigmoid

y D f sigmoid(r) D1

1 C eiexcl2b (riexcla)

where r is the activation (or ring rate) of the neuron after the lateral inhi-bition y is the ring rate after the contrast enhancement produced by theactivation function b is the slope or gain and a is the threshold or biasof the activation function The sigmoid bounds the ring rate between 0and 1 so global normalization is not required The slope and threshold areheld constant within each layer The slope is constant throughout trainingwhereas the threshold is used to control the sparseness of ring rates withineach layer The sparseness of the ring within a layer is dened (Rolls ampTreves 1998) as

a D(P

i yi n)2P

i y2i n

(213)

where n is the number of neurons in the layer To set the sparseness to agiven value for example 5 the threshold is set to the value of the 95thpercentile point of the activations within the layer (Unless otherwise statedhere the neurons used the sigmoid activation function as just described)

In this article we compare the results with short-range and longer-rangelateral inhibition and with the power and activation functions Unless oth-erwise stated the sigmoid activation function was used with parameters(selected after a number of optimization runs) as shown in Table 3 Thelateral inhibition parameters were as shown in Table 4 Where the poweractivation function was used the power for layer 1 was 6 and for the otherlayers was 2 (Wallis amp Rolls 1997)

24 Training and Test Procedure The network was trained by present-ing a stimulus in one training location calculating the activation of theneuron then its ring rate and then updating its synaptic weights Thesequence of locations for each stimulus was determined as follows For a


Table 4 Lateral Inhibition Parameters for 25 Location Runs

Layer 1 2 3 4

Radius s 138 27 40 60

Contrast d 15 15 16 14

Figure 3 (Left) Contiguous set of 25 training locations (Right) Five blocks ofve contiguous locations with random offset and direction

value of g of 08 the effect of the trace decays to 50 after 31 training loca-tions Therefore locations near the start of a xed sequence of for example25 locations would not occur in sufciently close temporal proximity forthe trace rule to work The images were therefore presented in sets of forexample 5 locations with then a jump to for example 5 more locationsuntil all 25 locations had been covered An analogy here is to a number ofsmall saccades (or alternatively small movements of the stimulus) punctu-ated by a larger saccade to another part of the stimulus Figure 3 shows theorder in which (for 25 locations) the locations were chosen by sequentiallytraversing blocks of 5 contiguous locations and visiting all 5 blocks in arandom order We also include random direction and offset componentsAfter all the training locations had been visited (25 unless otherwise statedfor the experiments described here) another stimulus was selected in a per-mutative sequence and the procedure was repeated Training in this wayfor all the stimuli in a set was one training epoch

The network was trained one layer at a time starting with layer 1 (abovethe retina) through to layer 4 The rationale for this was that there was nopoint in training a layer if the preceding layer had a continually changingrepresentation The number of epochs per layer were as shown in Table5 The learning rate was gradually decreasing to zero during the trainingof a layer according to a cosine function in the range 0 to p 2 in a form ofsimulated annealing The traceg value was set to 08 unless otherwise statedand for the runs described in this article the trace was articially reset to 0between different stimuli Each image was 64 pound64 pixels and was shown at


Table 5 Number of Training Epochs per Layer

Layer 1 2 3 4

Number of epochs 50 100 100 75

different positions in the 128 pound 128 ldquoretinardquo in the investigations describedin this article on translation invariance The number of pixels by whichthe image was translated was 32 for each move With a grid of 25 possiblelocations for the center of the image as shown in Figure 3 the maximumpossible shift of the center of the image was 64 pixels away from the centralpixel of the retina in both horizontal and vertical directions (see Figure 3)Wrap-around in the image plane ensured that images did not move off theldquoretinardquo The stimuli used (for both training and testing) were either thesame set of 7 faces used and illustrated in the previous investigations withVisNet (Wallis amp Rolls 1997) or an extended set of 17 faces

One training epoch consists of presenting each stimulus through each setof locations as described The trace was normally articially reset to zerobefore starting each stimulus presentation sequence but the effect of notresetting it was only mild as investigated here and described below

We now summarize the differences between VisNet and VisNet2 partlyto emphasize some of the new points addressed here and partly for clari-cation VisNet (see Wallis amp Rolls 1997) used a power activation functiononly short-range lateral inhibition the learning rule shown in equation 24primarily a Fisher measure of performance a linearly decreasing learningrate across the number of training trials used for a particular layer andtraining that always started at a xed position in the set of exemplars ofa stimulus VisNet2 uses a sigmoid activation function that incorporatescontrol of the sparseness of the representation and longer-range lateral in-hibition It allows comparison of the performance when trained with manyversions of a trace learning rule shown in equations 25 through 29 includesinformation theoryndashbased measures of performance has a learning rate thatdecreases with a cosine bell taper over approximately the last 20 of thetraining trials in an epoch and has training for a given stimulus that runsfor several short lengths of the exemplars of that stimulus each starting atrandom points among the exemplars in order to facilitate trace learningamong all exemplars of a given stimulus

3 Results

31 Comparison of the Fisher and Information Measures of NetworkPerformance The results of a run of VisNet with the standard parametersused in this article trained on 7 faces at each of 25 training locations areshown using the Fisher measure and the single-cell information measure in


0

5

10

15

20

25

10 20 30 40 50 60 70 80 90 100

Fis

her

Met

ric

Cell Rank

VisNet 7f 25l Single Cell Analysis

tracehebb

random

0

05

1

15

2

25

3

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


tracehebb

random

Fisher Metric Single Cell Information

Figure 4 Seven faces 25 locations Single cell analysis Fisher metric and In-formation measure for the best 100 cells Throughout this article the single cellinformation was calculated with equation 210

0

05

1

15

2

25

3

0 2 4 6 8 10 12 14 16 18

Info

rma

tion

(b

its)

Log Fisher Metric

Visnet 7f 25l Single Cell Analysis Information v Fisher

0

05

1

15

2

25

3

0 05 1 15 2 25 3 35

Info

rma

tion

(b

its)

Log Fisher Metric


Figure 5 Seven faces 25 locations Single cell analysis Comparison of the Fishermetric and the information measure The right panel shows an expanded partof the left panel

Figure 4 In both cases the values for the 100 most invariant cells are shown(ranked in order of their invariance) (Throughout this article the resultsfor the top layer designated 3 cells are shown The types of cell responsefound in lower layers are described by Wallis amp Rolls1997) It is clear that thenetwork performance is much better when trained with the trace rule thanwhen trained with a Hebb rule (which is not expected to capture invariances)or when left untrained with what is random synaptic connectivity Thisis indicated by both the Fisher and the single-cell information measuresFor further comparison we show in Figure 5 the information value fordifferent cells plotted against the log (because information is a log measure)of the Fisher measure It is evident that the two measures are closely relatedand indeed the correlation between them was 076 This means that theinformation measure can potentially replace the Fisher measure becausethey provide similar indications about the network performance

An advantage of the information measure is brought out next The in-formation about the most effective stimulus for the best cells is seen (seeFigure 4) to be in the region of 19 to 28 bits This information measure


0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index

Visnet 7f 25l Cell (29 31) Layer 3

rsquoface0rsquorsquoface1rsquorsquoface2rsquorsquoface3rsquorsquoface4rsquorsquoface5rsquorsquoface6rsquo

0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index



Information Percentage Correct

Figure 6 Seven faces 25 locations Response proles from two top layer cellsThe locations correspond to those shown in Figure 3 with the numbering fromtop left to bottom right as indicated

species what perfect performance by a cell would be if it had large re-sponses to one stimulus at all locations and small responses to all the otherstimuli at all locations The maximum value of the information I(s R) thatcould be provided by a cell in this way would be log2 NS D 281 bits Ex-amples of the responses of individual cells from this run are shown in Fig-ure 6 In these curves the different responses to a particular stimulus arethe responses at different locations If the responses to one stimulus didnot overlap at all with responses to the other stimuli the cell would giveunambiguous and invariant evidence about which stimulus was shown byits response on any one trial The cell in the right panel of Figure 6 diddiscriminate perfectly and its information value was 281 bits For the cellshown in the left panel of Figure 6 the responses to the different stimulioverlap a little and the information value was 213 bits The ability to in-terpret the actual value of the information measure that is possible in theway just shown is an advantage of the information measure relative to theFisher measure the exact value of which is difcult to interpret This to-gether with the fact that the information measure allows direct comparisonof the results with measures obtained from real neurons in neurophysio-logical experiments (Rolls amp Treves 1998 Tovee Rolls amp Azzopardi 1994Booth amp Rolls 1998) provides an advantage of the information measureand from now on we use the information measure instead of the Fishermeasure

We also note that the information measure is highly correlated (r D 093)with the index of invariance used for real neuronal data by Booth and Rolls(1998) The inverted form of this index (inverted to allow direct comparisonwith the information measure) is the variance of responses between stimulidivided by the variance of responses within the exemplars of a stimulus Inits inverted form it takes a high value if a neuron discriminateswell betweendifferent stimuli and has similar responses to the different exemplars of eachstimulus


0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells

VisNet 7f 25l Multiple Cell Analysis

tracehebb

random

0

20

40

60

80

100

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells


tracehebb

random


Figure 7 Seven faces 25 locations Multiple cell analysis Information and per-centage correct The multiple cell information was calculatedwith equation 212

Use of the multiple cell information measure I(S R) to quantify the per-formance of VisNet2 further is illustrated for the same run in Figure 7 (Amultiple cell assessment of the performance of this architecture has not beenperformed previously) It is shown that the multiple cell information risessteadily as the number of cells in the sample is increased with a plateaubeing reached at approximately 23 bits This indicates that the network canusing its 10 to 20 best cells in the output layer provide good but not per-fect evidence about which of the 7 stimuli has been shown (The measuredoes not reach the 28 bits that would indicate perfect performance mainlybecause not all stimuli had cells that coded perfectly for them) Consistentwith this the graph of percentage correctas a function of the number of cellsavailable from the output of the decoding algorithm shows performanceincreasing steadily but not quite reaching 100 Figure 7 also shows thatthere is moremultiple cell information available with the trace than with theHebb rule or when untrained showing correct operation of the trace rulein helping to form invariant representations The actual type of decodingused maximum likelihood or dot product does not make a great deal ofdifference to the values obtained (see Figure 8) so the maximum likelihooddecoding is used from now on We also use the multiple cell measure (andpercentage correct) from now on because there is no Fisher equivalent andbecause it is quantitative and shows whether the information required toidentify every stimulus in a set of a given size is available (log2 NS) becausethe percentage correct is available and because the value can be directlycompared with recordings from populations of real neurons (We note thatthe percentage correct is not expected to be linearly related to the infor-mation due to the metric structure of the space See Rolls amp Treves 1998section A234)

32 The Modied Trace Learning Rule A comparison of the differenttrace rules is shown in Figure 9 The major and very interesting performancedifference found was that for the three rules that calculate the synaptic


0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


trace opttrace dothebb opthebb dot

random opt random dot

10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells





Figure 8 Seven faces 25 locations Multiple cell analysis Information and per-centage correct Comparison of optimal (maximum likelihood) and dot productdecoding

0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank


pre (t)pre (t-1)both (t)

both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random

Single Cell Multiple Cell

Figure 9 Seven faces 25 locations Comparison of trace rules The three tracerules using the trace calculated at t iexcl1 (labeled as pre(tiexcl1) implementing equa-tion 27 post(t iexcl 1) implementing equation 25 and both(t iexcl 1) implementingequation 29) produced better performance than the three trace rules calculatedat t (labeled as pre(t) implementing equation 26 post(t) implementing equa-tion 24 and both(t) implementing equation 28) Pre- post- and both specifywhether the trace is present in the presynaptic or the postsynaptic terms or inboth

modication from the trace at time t iexcl 1 (see equations 25 27 and 29)the performance was much better than for the three rules that calculate thesynaptic modication from the trace at time t (see equations 24 26 and28) Within each of these two groups there was no difference between therule with the trace in the postsynaptic term (as used previously by Wallis ampRolls 1997) and the rules with the trace in the presynaptic term or in boththe presynaptic and postsynaptic term (Use of a trace of previous activitythat includes no contribution from the current activity has also been shownto be effective by Peng Sha Gan amp Wei 1998) As expected the performancewith no training that is with random weights was poor

The data shown in Figure 10 were for runs with a xed value ofg of 08 forlayers 2 through 4 Because the rules with the trace calculated at t iexcl1 might


0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l

no reset (t)reset (t)

no reset (t-1)reset (t-1)

Figure 10 Seven faces 25 locations Comparison of tracerules with and withouttrace reset and for various trace valuesg The performance measure used is themaximum multiple cell information available from the best 30 cells

have beneted from a slightly different value ofg from the rules using t weperformed further runs with different values of g for the two types of rulesWe show in Figure 10 that the better performance with the rules calculatingthe trace using t iexcl 1 occurred for a wide range of values of g Thus weconclude that the rules that calculate the trace using t iexcl 1 are indeed betterthan those that calculate it from t and that the difference is not due to anyinteraction with the value of g chosen

Unless otherwise stated the simulations in this article used a reset of thetrace to zero between different stimuli It is possible that in the brain thiscould be achieved by a mechanism associated with large eye movementsor it is possible that the brain does not have such a trace reset mechanism Inour earlier work (Wallis amp Rolls 1997) for generality we did not use tracereset To determine whether in practice this is important in the invariantrepresentations produced by VisNet in Figure 10 we explicitly compareperformance with and without trace reset for a range of different values ofg It was found that for most values of g (06ndash09) whether trace reset wasused made little difference For higher values ofg such as 095 trace reset didperform better The reason is that with a value ofg as high as 095 the effectof the trace without trace reset will last from the end of one stimulus well intothe presentations of the next stimulus thereby producing interference by theassociation together of two different stimuli Thus with 25 presentations ofeach stimulus trace reset is not necessary for good performance with valuesof g up to as high as 09 With 9 presentations of each stimulus we expectperformance with and without trace reset to be comparably good with avalue ofg up to 08 Further discussion of the optimal value and time courseof the trace is provided by Wallis and Baddeley (1997)

33 Activation Function and Lateral Inhibition The aim of this sectionis to compare performance with the sigmoid activation function introduced


for use with VisNet2 in this article with the power activation function asused previously (Wallis amp Rolls 1997) We demonstrate that the sigmoid ac-tivation function can produce better performance than the power activationfunction especially when the sparseness of the representation in each layeris kept at reasonably high (nonsparse) values As implemented it has theadditional advantage that the sparseness can be set and kept under precisecontrolWe also show that if the lateral inhibition operates over a reasonablylarge region of cells this can produce good performance In the previousstudy (Wallis amp Rolls 1997) only short-range lateral inhibition (in additionto the nonlinear power activation function) was used and the longer-rangelateral inhibition used here will help to ensure that different neurons withinthe wider region of lateral inhibition respond to different inputs This willhelp to reduce redundancy between nearby neurons

To apply the power activation function the powers of 6222 were usedfor layers 1 through 4 respectively as these values were found to be appro-priate and were used by Wallis and Rolls (1997) Although the nonlinearitypower for layer 1 needs to be high this is related to the need to reducethe sparseness below the very distributed representation produced by theinput layer of VisNet We aimed for sparseness in the region 001 through015 partly because sparseness in the brain is not usually lower than thisand partly because the aim of VisNet (1 and 2) is to represent each stimulusby a distributed representation in most layers so that neurons in higherlayers can learn about combinations of active neurons in a preceding layerAlthough making the representation extremely sparse in layer 1 with only1 of 1024 neurons active for each stimulus can produce very good perfor-mance this implies operation as a look-up table with each stimulus in eachposition represented by a different neuron This is not what the VisNet archi-tecture is intended to model Instead VisNet is intended to produce at anyone stage locally invariant representations of local feature combinationsAnother experimental advantage of the sigmoid activation function wasthat the sparseness could be set to particular values (using the percentileparameter) The parameters for the sigmoid activation function were as inTable 3

It is shown in Figure 11 that with limited numbers of stimuli (7) andtraining locations (9) performance is excellent with both the power and thesigmoid activation function With a more difcult problem of 17 faces and 25locations the sigmoid activation function produced better performance thanthe power activation function (see Figure 12) while at the same time using aless sparse representation than the power activation function (The averagesparseness for sigmoid versus power were layer 1 0008 versus 0002 layer2 002 versus 0004 layers 3 and 4 011 versus 0015) Similarly with the evenmore difcult problem of 17 faces and 49 locations the sigmoid activationfunction produced better performance than the power activation function(see Figure 13) while at the same time using a less sparse representationthan the power activation function similar to those just given


0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank

Visnet 7f 9l Single Cell Information

sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells

Visnet 7f 9l Multiple Cell Information

sigmoidpower


Figure 11 Seven faces 9 locations Comparison of the sigmoid and power acti-vation functions

05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower


Figure 12 Seventeen faces 25 locations Comparison of the sigmoid and poweractivation functions

The effects of different radii of lateral inhibition were most marked withdifcult training sets for example the 17 faces and 49 locations shown inFigure 14 Here it is shown that the best performancewas with intermediate-range lateral inhibition using the parameters for s shown in Table 4 Thesevalues of s are set so that the lateral inhibition radius within a layer is ap-proximately half that of the spread of the excitatory connections from the

0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank

Visnet 17f 49l Single Cell Analysis

narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells

Visnet 17f 49l Multiple Cell Analysis

narrowmedium

wide


Figure 14 Seventeen faces 49 locations Comparison of inhibitory mechanisms

preceding layer (For these runs only the activation function was binarythreshold The reason was to prevent the radius of lateral inhibition affect-ing the performance partly by interacting with the slope of the sigmoidactivation function) Using a small value for the radius s (set to one-fourthof the values given in Table 4) produced worse performance (presumablybecause similar feature analyzers can be formed quite nearby) This valueof the lateral inhibition radius is very similar to that used by Wallis andRolls (1997) Using a large value for the radius (set to four times the valuesgiven in Table 4) also produced worse performance (presumably becausethe inhibition was too global preventing all the local features from beingadequately represented) With simpler training sets (eg 7 faces and 25 lo-cations) the differences between the extents of lateral inhibition were lessmarked except that the more global inhibition again produced less goodperformance (not illustrated)

4 Discussion

We have introduced a more biologically realistic activation function for theneurons by replacing the power function with a sigmoid function The sig-moid function models not only the threshold nonlinearity of neurons butalso the fact that eventually the ring-rate of neurons saturates at a highrate (Although neurons cannot re for long at their maximal ring rate re-cent work on the ring-rate distribution of inferior temporal cortex neuronsshows that they only rarely reach high rates The ring-rate distribution hasan approximately exponential tail at high rates See Rolls Treves Tovee ampPanzeri 1997 and Treves Panzeri Rolls Booth amp Wakeman 1999) Theway in which the sigmoid was implemented did mean that some of theneurons would be close to saturation For example with the sigmoid biasa set at 95 approximately 4 of the neurons would be in the high ring-rate saturation region One advantage of the sigmoid was that it enabledthe sparseness to be accurately controlled A second advantage is that itenabled VisNet2 to achieve good performance with higher values of the


sparseness a than were possible with a power activation function VisNet(1 and 2) is of course intended to operate without very low values forsparseness The reason is that within every topologically organized regiondened for example as the size of the region in a preceding layer fromwhich a neuron receives inputs or as the area within a layer within whichcompetition operates some neurons should be active in order to representinformation within that topological region from the preceding layer Withthe power activation function a few neurons will tend to have high ringrates and given that normalization of activity is global across each layer ofVisNet the result may be that some topological regions have no neuronsactive A potential improvement to VisNet2 is thus to make the normaliza-tion of activity as well as the lateral inhibition operate separately withineach topological region set to be approximately the size of the region ofthe preceding layer from which a neuron receives its inputs (the connectionspread of Table 1) Another interesting difference between the sigmoid andthe power activation functions is that the distribution of ring rates withthe power activation function is monotonically decreasing with very fewneurons close to the highest rate In contrast the sigmoid activation func-tion tends to produce an almost binary distribution of ring rates with mostring rates 0 and for example 4 (set by the percentile) with rates close to1 It is this which helps VisNet to have at least some neurons active in eachof its topologically dened regions for each stimulus

Another design aspect of VisNet2 modeled on cortical architecturewhich is intended to produce some neurons in each topologically denedregion that are active for any stimulus is the lateral inhibition In the per-formance of VisNet analyzed previously (Wallis amp Rolls 1997) the lateralinhibition was set to a small radius of 1 pixel In the analyzes described hereit was shown as predicted that the operation of VisNet2 with longer-rangelateral inhibition was better with best performance with radii that wereapproximately the same as the region in the preceding layer from whicha neuron received excitatory inputs Using a smaller value for the radius(set to one-fourth of the normal values) produced worse performance (pre-sumably because similar feature analyzers can be formed quite nearby)Using a large value for the radius (set to four times the normal values) alsoproduced worse performance (presumably because the inhibition was tooglobal preventing all the local features from being adequately represented)

The investigation of the form of the trace rule showed that in VisNet2the use of a presynaptic trace a postsynaptic trace (as used previously) andboth presynaptic and postsynaptic traces produced similar performanceHowever a very interesting nding was that VisNet2 produced very muchbetter translation invariance if the rule used was with a trace calculatedfor t iexcl 1 for example equation 25 rather than at time t for exampleequation 24 (see Figure 9) One way to understand this is to note that thetrace rule is trying to set up the synaptic weight on trial t based on whetherthe neuron based on its previous history is responding to that stimulus (in


other positions) Use of the trace rule at t iexcl 1 does this that is takes intoaccount the ring of the neuron on previous trials with no contributionfrom the ring being produced by the stimulus on the current trial Onthe other hand use of the trace at time t in the update takes into accountthe current ring of the neuron to the stimulus in that particular positionwhich is not a good estimate of whether that neuron should be allocatedto represent that stimulus invariantly Effectively using the trace at time t

introduces a Hebbian element into the update which tends to build position-encoded analyzers rather than stimulus-encoded analyzers (The argumenthas been phrased for a system learning translation invariance but appliesto the learning of all types of invariance) A particular advantage of usingthe trace at t iexcl 1 is that the trace will then on different occasions (due to therandomness in the location sequences used) reect previous histories withdifferent sets of positions enabling the learning of the neuron to be basedon evidence from the stimulus present in many different positions Usinga term from the current ring in the trace (the trace calculated at time t )results in this desirable effect always having an undesirable element fromthe current ring of the neuron to the stimulus in its current position

This discussion of the trace rule shows that a good way to update weightsin VisNet is to use evidence from previous trials but not the current trialThis may be compared with the powerful methods that use temporal dif-ference (TD) learning (Sutton 1988 Sutton amp Barto 1998) In TD learningthe synaptic weight update is based on the difference between the currentestimate at time t and the previous estimate available at t iexcl 1 This is aform of error correction learning rather than the associative use of a traceimplemented in VisNet for biological plausibility If cast in the notation ofVisNet a TD(l) rule might appear as

Dwj (yt iexcl yt iexcl1)xt iexcl1j (41)

dt xt iexcl1j (42)

where dt D yt iexcl yt iexcl1 is the difference between the current ring rate andthe previous ring rate This latter term can be thought of as the error of theprediction by the neuron of which stimulus is present An alternative TD(0)rule might appear as


dtxt iexcl1

j (44)

where dt

is a trace of dt the difference between the current ring rate andthe previous ring rate The interesting similarity between the trace rule att iexcl 1 and the TD rule is that neither updates the weights using a Hebbiancomponent that is based on the ring at time t of both the presynaptic


and postsynaptic terms It will be of interest in future work to investigatehow much better VisNet may performif an error-based synaptic update rulerather than the trace rule is used However we note that holding a trace fromprevious trials or using the activity from previous trials to help generate anerror for TD learning is not immediately biologically plausible without aspecial implementation and the fact that VisNet2 can operate well with theordinary trace rules (equations at time t) is important and differentiates itfrom TD learning

The results described here have shown that VisNet2 can perform reason-ably when set more difcult problems than those on which it has been testedpreviously by Wallis and Rolls (1997) In particular the number of trainingand test locations was increased from 9 to 25 and 49 In order to allow thetrace rule to learn over this many variations in the translation of the imagesa method of presenting images with many small and some larger excursionswas introduced The rationale we have in mind is that eye movements aswell as movements of objects could produce the appropriate movementsof objects across the retina so that the trace rule can help the learning ofinvariant representations of objects In addition in the work described herefor some simulations the number of face stimuli was increased from 7 to17 Although performance was not perfect with very many different stim-uli (17) and very many different locations (49) we note that this particularproblem is hard and that ways to increase the capacity (such as limiting thesparseness in the later layers) have not been the subject of the investigationsdescribed here We also note that an analytic approach to the capacity of anetwork with a different architecture in providing invariant representationshas been developed (Parga amp Rolls 1998)

A comparison of VisNet with other architectures for producing invariantrepresentations of objects is provided elsewhere (Wallis amp Rolls 1997 Pargaamp Rolls 1998 see also Bartlett amp Sejnowski 1997 Stone 1996 Salinas ampAbbott 1997) The results by extending VisNet to use sigmoid activationfunctions that enable the sparseness of the representation to be set accuratelyshowing that a more appropriate range of lateral inhibition can improveperformance as predicted introducing new measures of its performanceand the discovery of an improved type of trace learning rule have provideduseful further evidence that the VisNet architecture does capture and helpto explore some aspects of invariant visual object recognition as performedby the primate visual system

Acknowledgments

The research was supported by the Medical Research Council grantPG8513790 to E T R The authors acknowledge the valuable contribution ofGuy Wallis who worked on an earlier version of VisNet (see Wallis amp Rolls1997) and are grateful to Roland Baddeley of the MRC InterdisciplinaryResearch Center for Cognitive Neuroscience at Oxford for many helpful


comments The authors thank Martin Elliffe for assistance with code to pre-pare the output of VisNet2 for input to the information-theoretic routinesdeveloped and described by Rolls Treves Tovee and Panzeri (1997) andRolls Treves and Tovee (1997) Stefano Panzeri for providing equispacedbins for the multiple cell information analysis routines developed by RollsTreves and Tovee (1997) and Simon Stringer for performing the compar-ison of the information measure with the invariance index used by Boothand Rolls (1998)

References

Bartlett M S amp Sejnowski T J (1997)Viewpoint invariant face recognition us-ing independent component analysis and attractor networks In M MozerM Jordan amp T Petsche (Eds) Advances in neural information processing sys-tems 9 Cambridge MA MIT Press

Booth M C A amp Rolls E T (1998) View-invariant representations of familiarobjects by neurons in the inferior temporal visual cortex Cerebral Cortex 8510ndash523

Desimone R (1991) Face-selective cells in the temporal cortex of monkeysJournal of Cognitive Neuroscience 3 1ndash8

Foldiak P (1991) Learning invariance from transformation sequences NeuralComputation 3 194ndash200

Hawken M J amp Parker A J (1987) Spatial properties of the monkey striatecortex Proceedings of the Royal Society London [B] 231 251ndash288

Panzeri S Treves A SchultzSamp Rolls E T (1999)On decoding the responsesof a population of neurons from short time windows Neural Computation 111553ndash1577

Parga N amp Rolls E T (1998) Transform invariant recognition by associationin a recurrent network Neural Computation 10 1507ndash1525

Peng H C Sha L F Gan Q amp Wei Y (1998) Energy function for learninginvariance in multilayer perceptron Electronics Letters 34(3) 292ndash294

Rolls E T (1992) Neurophysiological mechanisms underlying face processingwithin and beyond the temporal cortical areas Philosophical Transactions ofthe Royal Society London [B] 335 11ndash21

Rolls E T (1994) Brain mechanisms for invariant visual recognition and learn-ing Behavioural Processes 33 113ndash138

Rolls E T (1995) Learning mechanisms in the temporal lobe visual cortexBehavioural Brain Research 66 177ndash185

Rolls E T (1997) A neurophysiological and computational approach to thefunctions of the temporal lobe cortical visual areas in invariant object recog-nition In L Harris amp M Jenkin (Eds) Computational and psychophysicalmecha-nisms of visual coding (pp 184ndash220)Cambridge Cambridge University Press

Rolls E T (2000) Functions of the primate temporal lobe cortical visual areasin invariant visual object and face recognition Neuron 27 1ndash20


Rolls E T amp Tovee M J (1995) Sparseness of the neuronal representation ofstimuli in the primate temporal visual cortex Journal of Neurophysiology 73713ndash726

Rolls E T amp Treves A (1998)Neural networksand brain function Oxford OxfordUniversity Press

Rolls E T Treves A amp Tovee M J (1997)The representational capacity of thedistributed encoding of information provided by populations of neurons inthe primate temporal visual cortex Experimental Brain Research 114 177ndash185

Rolls E TTreves A Tovee Mamp Panzeri S (1997)Information in the neuronalrepresentation of individual stimuli in the primate temporal visual cortexJournal of Computational Neuroscience 4 309ndash333

Salinas E amp Abbott L F (1997) Invariant visual responses from attentionalgain elds Journal of Neurophysiology77 3267ndash3272

Stone J V (1996)A canonicalmicrofunction for learning perceptual invariancesPerception 25 207ndash220

Sutton R S (1988)Learning to predict by the methods of temporal differencesMachine Learning 3 9ndash44

Sutton R S amp Barto A G (1981) Towards a modern theory of adaptive net-works Expectation and prediction Psychological Review 88 135ndash170

Sutton R S amp Barto A G (1998)Reinforcement learning Cambridge MA MITPress

Tanaka K Saito H Fukada Y amp Moriya M (1991) Coding visual imagesof objects in the inferotemporal cortex of the macaque monkey Journal ofNeurophysiology 66 170ndash189

Tovee M J Rolls E T amp Azzopardi P (1994) Translation invariance and theresponses of neurons in the temporal visual cortical areas of primates Journalof Neurophysiology 72 1049ndash1060

Treves A Panzeri S Rolls E T Booth M amp Wakeman E A (1999)Firing ratedistributions and efciency of information transmission of inferior temporalcortex neurons to natural visual stimuli Neural Computation 11 611ndash641

Wallis G amp Baddeley R (1997) Optimal unsupervised learning in invari-ant object recognition Neural Computation 9(4) 959ndash970 Available onlineat ftpftpmpik-tuebmpgdepubguyncps

Wallis G Rolls E T (1997)A model of invariant object recognition in the visualsystem Progress in Neurobiology 51 167ndash194

Wallis G Rolls E T ampFoldiak P (1993) Learning invariant responses to thenatural transformations of objects International Joint Conference on NeuralNet-works 2 1087ndash1090

Accepted April 8 1999 received November 24 1998


(Desimone 1991 Rolls 1992 Rolls amp Tovee 1995 Tanaka Saito Fukadaamp Moriya 1991) (see Figure 2) Rolls (1992 1994 1995 1997 2000) has re-viewed much of this neurophysiological work and has advanced a theoryfor how these neurons could acquire their transform-independent selectiv-ity based on the known physiology of the visual cortex and self-organizingprinciples (see also Wallis amp Rolls 1997 Rolls amp Treves 1998)

Rollsrsquos hypothesis has the following fundamental elements

A series of competitive networks organized in hierarchical layers ex-hibiting mutual inhibition over a short range within each layer Thesenetworks allow combinations of features or inputs that occur in a givenspatial arrangement to be learned by neurons ensuring that higher-order spatial properties of the input stimuli are represented in thenetwork

A convergent series of connections from a localized population of cellsin preceding layers to each cell of the following layer thus allowing thereceptive-eld size of cells to increase through the visual processingareas or layers

A modied Hebb-like learning rule incorporating a temporal trace ofeach cellrsquos previous activity which it is suggested will enable theneurons to learn transform invariances (see also Foldiak 1991)

Based on these hypotheses Wallis and Rolls produced a model of ventralstream cortical visual processing designed to investigate computational as-pects of this processing (Wallis Rolls amp Foldiak 1993 Wallis amp Rolls 1997)With this model it has been shown that provided that a trace learning ruleto be dened below is used the network (VisNet) can learn transform-invariant representations It can produce neurons that respond to some butnot other stimuli with translation and view invariance In investigations oftranslation invariance described by Wallis and Rolls (1997) the stimuli weresimple stimuli such as T L and C or more complex stimuli such as facesRelatively few (up to seven) different stimuli such as different faces wereused for training

The work described here investigates several key issues that inuencehow the network operates including new formulations of the trace rule thatsignicantly improve the performance of the network how local inhibitionwithin a layer may improve the performance of the network how neuronswith a sigmoid activation function that allows the sparseness of the repre-sentation to be controlled can improve the performance of the network andhow the performance of the network scales up when a larger number oftraining locations and a larger number of stimuli are used The model de-scribed previously by Wallis and Rolls (1997) was modied in the ways justdescribed to produce the model described here which is denoted VisNet2

Before describing these new results we rst outline the architecture ofthe model In section 2 we describe a way to measure the performance of





Dwj D ayt xj

where









Rec

eptiv

e Fi

eld

Siz

e d

eg

Layer 4

Layer 3

Layer 2

Layer 1









50

0 3213 80 20 50

20

TE

TEO

V4

V2

V1

LGN



Rec

epti

ve F

ield

Siz

e d

eg

Eccentricity deg


view independence

80

32

13







Frequency 00625 0125 025 05




Pi rp



2 Methods






y D fplusmnX

wjxj

sup2(21)

xtj D (1 iexcl g)xt



Dw1j yt xt

j (24)

Dw2j yt iexcl1xt

j (25)

Dw3j yt xt

j (26)

Dw4j yt xt iexcl1

j (27)

Dw5j yt xt

j C yt xtj D Dw1

j C Dw3j (28)

Dw6j yt iexcl1xt


j C Dw4j (29)




I(s R) DX

r2R


P(r) (210)







I(S R) DX

s2S

P(s)I(s R) (211)




I(S S0 ) DX

ss0


P(s)P(s0 ) (212)




Iab D

8gtltgt



1 iexclP

a 6D0b 6D0





Layer 1 2 3 4


Slope b 190 40 75 26


y D f sigmoid(r) D1



a D(P

i yi n)2P

i y2i n

(213)






Layer 1 2 3 4

Radius s 138 27 40 60







Layer 1 2 3 4





3 Results



0

5

10

15

20

25

10 20 30 40 50 60 70 80 90 100

Fis

her

Met

ric

Cell Rank


tracehebb

random

0

05

1

15

2

25

3

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


tracehebb

random



0

05

1

15

2

25

3

0 2 4 6 8 10 12 14 16 18

Info

rma

tion

(b

its)

Log Fisher Metric


0

05

1

15

2

25

3

0 05 1 15 2 25 3 35

Info

rma

tion

(b

its)

Log Fisher Metric






0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index



0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index








0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


tracehebb

random

0

20

40

60

80

100

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells


tracehebb

random






0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells




10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells






0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank



both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random






0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References



































Dwj D ayt xj

where









Rec

eptiv

e Fi

eld

Siz

e d

eg

Layer 4

Layer 3

Layer 2

Layer 1









50

0 3213 80 20 50

20

TE

TEO

V4

V2

V1

LGN



Rec

epti

ve F

ield

Siz

e d

eg

Eccentricity deg


view independence

80

32

13







Frequency 00625 0125 025 05




Pi rp



2 Methods






y D fplusmnX

wjxj

sup2(21)

xtj D (1 iexcl g)xt



Dw1j yt xt

j (24)

Dw2j yt iexcl1xt

j (25)

Dw3j yt xt

j (26)

Dw4j yt xt iexcl1

j (27)

Dw5j yt xt

j C yt xtj D Dw1

j C Dw3j (28)

Dw6j yt iexcl1xt


j C Dw4j (29)




I(s R) DX

r2R


P(r) (210)







I(S R) DX

s2S

P(s)I(s R) (211)




I(S S0 ) DX

ss0


P(s)P(s0 ) (212)




Iab D

8gtltgt



1 iexclP

a 6D0b 6D0





Layer 1 2 3 4


Slope b 190 40 75 26


y D f sigmoid(r) D1



a D(P

i yi n)2P

i y2i n

(213)






Layer 1 2 3 4

Radius s 138 27 40 60







Layer 1 2 3 4





3 Results



0

5

10

15

20

25

10 20 30 40 50 60 70 80 90 100

Fis

her

Met

ric

Cell Rank


tracehebb

random

0

05

1

15

2

25

3

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


tracehebb

random



0

05

1

15

2

25

3

0 2 4 6 8 10 12 14 16 18

Info

rma

tion

(b

its)

Log Fisher Metric


0

05

1

15

2

25

3

0 05 1 15 2 25 3 35

Info

rma

tion

(b

its)

Log Fisher Metric






0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index



0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index








0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


tracehebb

random

0

20

40

60

80

100

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells


tracehebb

random






0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells




10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells






0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank



both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random






0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References
































Rec

eptiv

e Fi

eld

Siz

e d

eg

Layer 4

Layer 3

Layer 2

Layer 1









50

0 3213 80 20 50

20

TE

TEO

V4

V2

V1

LGN



Rec

epti

ve F

ield

Siz

e d

eg

Eccentricity deg


view independence

80

32

13







Frequency 00625 0125 025 05




Pi rp



2 Methods






y D fplusmnX

wjxj

sup2(21)

xtj D (1 iexcl g)xt



Dw1j yt xt

j (24)

Dw2j yt iexcl1xt

j (25)

Dw3j yt xt

j (26)

Dw4j yt xt iexcl1

j (27)

Dw5j yt xt

j C yt xtj D Dw1

j C Dw3j (28)

Dw6j yt iexcl1xt


j C Dw4j (29)




I(s R) DX

r2R


P(r) (210)







I(S R) DX

s2S

P(s)I(s R) (211)




I(S S0 ) DX

ss0


P(s)P(s0 ) (212)




Iab D

8gtltgt



1 iexclP

a 6D0b 6D0





Layer 1 2 3 4


Slope b 190 40 75 26


y D f sigmoid(r) D1



a D(P

i yi n)2P

i y2i n

(213)






Layer 1 2 3 4

Radius s 138 27 40 60







Layer 1 2 3 4





3 Results



0

5

10

15

20

25

10 20 30 40 50 60 70 80 90 100

Fis

her

Met

ric

Cell Rank


tracehebb

random

0

05

1

15

2

25

3

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


tracehebb

random



0

05

1

15

2

25

3

0 2 4 6 8 10 12 14 16 18

Info

rma

tion

(b

its)

Log Fisher Metric


0

05

1

15

2

25

3

0 05 1 15 2 25 3 35

Info

rma

tion

(b

its)

Log Fisher Metric






0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index



0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index








0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


tracehebb

random

0

20

40

60

80

100

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells


tracehebb

random






0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells




10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells






0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank



both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random






0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References



































50

0 3213 80 20 50

20

TE

TEO

V4

V2

V1

LGN



Rec

epti

ve F

ield

Siz

e d

eg

Eccentricity deg


view independence

80

32

13







Frequency 00625 0125 025 05




Pi rp



2 Methods






y D fplusmnX

wjxj

sup2(21)

xtj D (1 iexcl g)xt



Dw1j yt xt

j (24)

Dw2j yt iexcl1xt

j (25)

Dw3j yt xt

j (26)

Dw4j yt xt iexcl1

j (27)

Dw5j yt xt

j C yt xtj D Dw1

j C Dw3j (28)

Dw6j yt iexcl1xt


j C Dw4j (29)




I(s R) DX

r2R


P(r) (210)







I(S R) DX

s2S

P(s)I(s R) (211)




I(S S0 ) DX

ss0


P(s)P(s0 ) (212)




Iab D

8gtltgt



1 iexclP

a 6D0b 6D0





Layer 1 2 3 4


Slope b 190 40 75 26


y D f sigmoid(r) D1



a D(P

i yi n)2P

i y2i n

(213)






Layer 1 2 3 4

Radius s 138 27 40 60







Layer 1 2 3 4





3 Results



0

5

10

15

20

25

10 20 30 40 50 60 70 80 90 100

Fis

her

Met

ric

Cell Rank


tracehebb

random

0

05

1

15

2

25

3

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


tracehebb

random



0

05

1

15

2

25

3

0 2 4 6 8 10 12 14 16 18

Info

rma

tion

(b

its)

Log Fisher Metric


0

05

1

15

2

25

3

0 05 1 15 2 25 3 35

Info

rma

tion

(b

its)

Log Fisher Metric






0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index



0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index








0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


tracehebb

random

0

20

40

60

80

100

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells


tracehebb

random






0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells




10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells






0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank



both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random






0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References

































Frequency 00625 0125 025 05




Pi rp



2 Methods






y D fplusmnX

wjxj

sup2(21)

xtj D (1 iexcl g)xt



Dw1j yt xt

j (24)

Dw2j yt iexcl1xt

j (25)

Dw3j yt xt

j (26)

Dw4j yt xt iexcl1

j (27)

Dw5j yt xt

j C yt xtj D Dw1

j C Dw3j (28)

Dw6j yt iexcl1xt


j C Dw4j (29)




I(s R) DX

r2R


P(r) (210)







I(S R) DX

s2S

P(s)I(s R) (211)




I(S S0 ) DX

ss0


P(s)P(s0 ) (212)




Iab D

8gtltgt



1 iexclP

a 6D0b 6D0





Layer 1 2 3 4


Slope b 190 40 75 26


y D f sigmoid(r) D1



a D(P

i yi n)2P

i y2i n

(213)






Layer 1 2 3 4

Radius s 138 27 40 60







Layer 1 2 3 4





3 Results



0

5

10

15

20

25

10 20 30 40 50 60 70 80 90 100

Fis

her

Met

ric

Cell Rank


tracehebb

random

0

05

1

15

2

25

3

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


tracehebb

random



0

05

1

15

2

25

3

0 2 4 6 8 10 12 14 16 18

Info

rma

tion

(b

its)

Log Fisher Metric


0

05

1

15

2

25

3

0 05 1 15 2 25 3 35

Info

rma

tion

(b

its)

Log Fisher Metric






0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index



0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index








0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


tracehebb

random

0

20

40

60

80

100

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells


tracehebb

random






0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells




10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells






0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank



both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random






0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References



































y D fplusmnX

wjxj

sup2(21)

xtj D (1 iexcl g)xt



Dw1j yt xt

j (24)

Dw2j yt iexcl1xt

j (25)

Dw3j yt xt

j (26)

Dw4j yt xt iexcl1

j (27)

Dw5j yt xt

j C yt xtj D Dw1

j C Dw3j (28)

Dw6j yt iexcl1xt


j C Dw4j (29)




I(s R) DX

r2R


P(r) (210)







I(S R) DX

s2S

P(s)I(s R) (211)




I(S S0 ) DX

ss0


P(s)P(s0 ) (212)




Iab D

8gtltgt



1 iexclP

a 6D0b 6D0





Layer 1 2 3 4


Slope b 190 40 75 26


y D f sigmoid(r) D1



a D(P

i yi n)2P

i y2i n

(213)






Layer 1 2 3 4

Radius s 138 27 40 60







Layer 1 2 3 4





3 Results



0

5

10

15

20

25

10 20 30 40 50 60 70 80 90 100

Fis

her

Met

ric

Cell Rank


tracehebb

random

0

05

1

15

2

25

3

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


tracehebb

random



0

05

1

15

2

25

3

0 2 4 6 8 10 12 14 16 18

Info

rma

tion

(b

its)

Log Fisher Metric


0

05

1

15

2

25

3

0 05 1 15 2 25 3 35

Info

rma

tion

(b

its)

Log Fisher Metric






0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index



0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index








0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


tracehebb

random

0

20

40

60

80

100

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells


tracehebb

random






0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells




10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells






0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank



both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random






0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References

































I(s R) DX

r2R


P(r) (210)







I(S R) DX

s2S

P(s)I(s R) (211)




I(S S0 ) DX

ss0


P(s)P(s0 ) (212)




Iab D

8gtltgt



1 iexclP

a 6D0b 6D0





Layer 1 2 3 4


Slope b 190 40 75 26


y D f sigmoid(r) D1



a D(P

i yi n)2P

i y2i n

(213)






Layer 1 2 3 4

Radius s 138 27 40 60







Layer 1 2 3 4





3 Results



0

5

10

15

20

25

10 20 30 40 50 60 70 80 90 100

Fis

her

Met

ric

Cell Rank


tracehebb

random

0

05

1

15

2

25

3

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


tracehebb

random



0

05

1

15

2

25

3

0 2 4 6 8 10 12 14 16 18

Info

rma

tion

(b

its)

Log Fisher Metric


0

05

1

15

2

25

3

0 05 1 15 2 25 3 35

Info

rma

tion

(b

its)

Log Fisher Metric






0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index



0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index








0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


tracehebb

random

0

20

40

60

80

100

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells


tracehebb

random






0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells




10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells






0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank



both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random






0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References


































I(S R) DX

s2S

P(s)I(s R) (211)




I(S S0 ) DX

ss0


P(s)P(s0 ) (212)




Iab D

8gtltgt



1 iexclP

a 6D0b 6D0





Layer 1 2 3 4


Slope b 190 40 75 26


y D f sigmoid(r) D1



a D(P

i yi n)2P

i y2i n

(213)






Layer 1 2 3 4

Radius s 138 27 40 60







Layer 1 2 3 4





3 Results



0

5

10

15

20

25

10 20 30 40 50 60 70 80 90 100

Fis

her

Met

ric

Cell Rank


tracehebb

random

0

05

1

15

2

25

3

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


tracehebb

random



0

05

1

15

2

25

3

0 2 4 6 8 10 12 14 16 18

Info

rma

tion

(b

its)

Log Fisher Metric


0

05

1

15

2

25

3

0 05 1 15 2 25 3 35

Info

rma

tion

(b

its)

Log Fisher Metric






0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index



0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index








0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


tracehebb

random

0

20

40

60

80

100

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells


tracehebb

random






0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells




10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells






0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank



both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random






0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References

































I(S S0 ) DX

ss0


P(s)P(s0 ) (212)




Iab D

8gtltgt



1 iexclP

a 6D0b 6D0





Layer 1 2 3 4


Slope b 190 40 75 26


y D f sigmoid(r) D1



a D(P

i yi n)2P

i y2i n

(213)






Layer 1 2 3 4

Radius s 138 27 40 60







Layer 1 2 3 4





3 Results



0

5

10

15

20

25

10 20 30 40 50 60 70 80 90 100

Fis

her

Met

ric

Cell Rank


tracehebb

random

0

05

1

15

2

25

3

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


tracehebb

random



0

05

1

15

2

25

3

0 2 4 6 8 10 12 14 16 18

Info

rma

tion

(b

its)

Log Fisher Metric


0

05

1

15

2

25

3

0 05 1 15 2 25 3 35

Info

rma

tion

(b

its)

Log Fisher Metric






0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index



0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index








0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


tracehebb

random

0

20

40

60

80

100

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells


tracehebb

random






0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells




10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells






0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank



both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random






0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References

































Layer 1 2 3 4


Slope b 190 40 75 26


y D f sigmoid(r) D1



a D(P

i yi n)2P

i y2i n

(213)






Layer 1 2 3 4

Radius s 138 27 40 60







Layer 1 2 3 4





3 Results



0

5

10

15

20

25

10 20 30 40 50 60 70 80 90 100

Fis

her

Met

ric

Cell Rank


tracehebb

random

0

05

1

15

2

25

3

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


tracehebb

random



0

05

1

15

2

25

3

0 2 4 6 8 10 12 14 16 18

Info

rma

tion

(b

its)

Log Fisher Metric


0

05

1

15

2

25

3

0 05 1 15 2 25 3 35

Info

rma

tion

(b

its)

Log Fisher Metric






0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index



0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index








0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


tracehebb

random

0

20

40

60

80

100

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells


tracehebb

random






0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells




10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells






0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank



both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random






0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References

































Layer 1 2 3 4

Radius s 138 27 40 60







Layer 1 2 3 4





3 Results



0

5

10

15

20

25

10 20 30 40 50 60 70 80 90 100

Fis

her

Met

ric

Cell Rank


tracehebb

random

0

05

1

15

2

25

3

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


tracehebb

random



0

05

1

15

2

25

3

0 2 4 6 8 10 12 14 16 18

Info

rma

tion

(b

its)

Log Fisher Metric


0

05

1

15

2

25

3

0 05 1 15 2 25 3 35

Info

rma

tion

(b

its)

Log Fisher Metric






0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index



0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index








0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


tracehebb

random

0

20

40

60

80

100

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells


tracehebb

random






0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells




10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells






0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank



both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random






0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References

































Layer 1 2 3 4





3 Results



0

5

10

15

20

25

10 20 30 40 50 60 70 80 90 100

Fis

her

Met

ric

Cell Rank


tracehebb

random

0

05

1

15

2

25

3

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


tracehebb

random



0

05

1

15

2

25

3

0 2 4 6 8 10 12 14 16 18

Info

rma

tion

(b

its)

Log Fisher Metric


0

05

1

15

2

25

3

0 05 1 15 2 25 3 35

Info

rma

tion

(b

its)

Log Fisher Metric






0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index



0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index








0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


tracehebb

random

0

20

40

60

80

100

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells


tracehebb

random






0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells




10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells






0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank



both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random






0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References
































0

5

10

15

20

25

10 20 30 40 50 60 70 80 90 100

Fis

her

Met

ric

Cell Rank


tracehebb

random

0

05

1

15

2

25

3

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


tracehebb

random



0

05

1

15

2

25

3

0 2 4 6 8 10 12 14 16 18

Info

rma

tion

(b

its)

Log Fisher Metric


0

05

1

15

2

25

3

0 05 1 15 2 25 3 35

Info

rma

tion

(b

its)

Log Fisher Metric






0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index



0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index








0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


tracehebb

random

0

20

40

60

80

100

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells


tracehebb

random






0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells




10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells






0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank



both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random






0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References
































0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index



0

02

04

06

08

1

5 10 15 20

Firi

ng R

ate

Location Index








0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


tracehebb

random

0

20

40

60

80

100

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells


tracehebb

random






0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells




10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells






0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank



both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random






0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References
































0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


tracehebb

random

0

20

40

60

80

100

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells


tracehebb

random






0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells




10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells






0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank



both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random






0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References
































0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells




10

20

30

40

50

60

70

80

90

5 10 15 20 25 30

Per

cent

age

Co

rrec

t

Number of Cells






0

05

1

15

2

25

3

20 40 60 80 100 120 140 160

Info

rma

tion

(b

its)

Cell Rank



both (t-1)post (t)

post (t-1)random

0

05

1

15

2

25

10 20 30 40 50

Info

rma

tion

(b

its)

Number of Cells



both (t-1)post (t)

post (t-1)random






0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References
































0608

112141618

2222426

055 06 065 07 075 08 085 09 095 1

Max

Info

(bi

ts)

Trace Eta

Visnet 7f 25l












0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References




































0

05

1

15

2

25

3

0 100 200 300 400 500 600 700 800 900

Info

rmat

ion

(b

its)

Cell Rank


sigmoidpower

05

1

15

2

25

3

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


sigmoidpower



05

1

15

2

25

3

35

4

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower

0

05

1

15

2

25

5 10 15 20 25 30

Info

rmat

ion

(bits

)

Cell Rank


sigmoidpower




0

05

1

15

2

25

10 20 30 40 50 60 70 80 90 100

Info

rma

tion

(b

its)

Cell Rank


sigmoidpower

0

02

04

06

08

1

12

14

5 10 15 20 25 30

Info

rma

tion

(b

its)

Number of Cells


sigmoidpower




04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References
































04

06

08

1

12

14

16

18

10 20 30 40 50 60 70 80 90 100

Info

rmat

ion

(b

its)

Cell Rank


narrowmedium

wide

0010203040506070809

1

5 10 15 20 25 30

Info

rmat

ion

(b

its)

Number of Cells


narrowmedium

wide




4 Discussion











dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References








































dt xt iexcl1j (42)



dtxt iexcl1

j (44)

where dt






Acknowledgments




References



































Acknowledgments




References

































References































a model of invariant object recognition in the visual ...milward00.pdf · invariant object...

Documents