cmos implementation of an artificial neuron training on

CMOS Implementation of an Artificial Neuron Training on Logical Threshold Functions

V. VARSHAVSKY, V. MARAKHOVSKY1, H. SAITO2

1Saint-Petersburg State Polytechnic University, Polytechnicheskaya str., 29, Saint Petersburg,

RUSSIA 2 The University of Aizu

Aizu-Wakamatsu City, Fukushima Prefecture, 965-8580 JAPAN

[email protected], [email protected] Abstract: - This paper offers a new methodology for designing in CMOS technology analog-digital artificial neurons training on arbitrary logical threshold functions of some number of variables. The problems of functional ability, implementability restrictions, noise stability, and refreshment of the learned state are formulated and solved. Some functional problems in experiments on teaching logical functions to an artificial neuron are considered. Recommendations are given on selecting testing functions and generating teaching sequences. All results in the paper are obtained using SPICE simulation. For simulation experiments with analog/digital CMOS circuits, transistor models MOSIS BSIM3v3.1, 0.8µm, level 7 are used. Key-Words: - Artificial neuron, CMOS implementation, learnable synapse, excitatory and inhibitory inputs, learning process, learning sequence, refreshment process, test function, threshold logical element, threshold logical function, Horner's scheme, Fibonacci sequence. 1 Introduction Hardware implementation of an artificial neuron has a number of well-known advantages over software implementation [1–5]. The hardware implementation of an artificial neuron can take the form of either a special purpose programmable controller or digital/analog circuit (device). Each type of implementations has its advantages, drawbacks, and fields of application. Although analog/digital implementation has the advantage of high performance, there are rigid limitations on the class of realizable threshold functions due to its analog nature. These limitations considerably decrease the functional possibilities of neural nets that have a fixed number of neurons.

The functional power of a neurochip depends equally on the number of neurons that can be placed on one VLSI and the functional possibilities of a single neuron. Unfortunately, the effects of these parameters on the functional power of the neurochip have not been studied. However, before creating new neurochips, it is necessary to decrease the area/synapse and extend the functional possibilities of a neuron.

In [6, 7], a new type of threshold element (β-driven threshold element, β-DTE) was offered that

required one transistor per logical input. Its circuit was based on representing a threshold function in ratio form. In [8–11], a CMOS learnable neuron was proposed on the base of β-DTE that consisted of synapses, a β-comparator, and an output amplifier. The learnable synapse of this neuron had five transistors and one capacitor. The neuron had one remarkable property: its implementability depended only on the threshold value and not on the number of logical inputs or their weights. This fact coupled with its relatively low complexity made this neuron very attractive for use in the next generation of digital-analog neurochips.

An artificial neuron designed for implementation of logical threshold functions it is more correctly called a learnable threshold element (LTE). During learning, this device creates analog weights for binary (digital) input variables. Obviously, an actual artificial neuron can be constructed based on LTE.

The goal of this paper is to improve the LTE circuit in terms of its learnability for complicated logical threshold functions (with a large value of the minimum threshold), noise-stability, and ability to maintain the learned state for a long time.

When the function threshold is high, the noise-stability becomes especially important. It is

WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS V. Varshavsky, V. Marakhovsky, H. Saito

ISSN: 1109-2734 370 Issue 4, Volume 8, April 2009

dvtttit

iLacaLaaaitttwta

plmuhclpilfptommfccfemtumwlnp

determined bvoltage minthreshold. Lathe sharpnesthe threshoincorporatingtransistors an

The noimplementabLTE dependalso on the tcharacteristicamplifier. ThLTE a given allows not amplifier thralso increaseis attained wthe function the β-compathree outputwhich providthis hysteresattained durin

Some adpaper. One plearned statememory on cuses the sahysteresis ofconcerns thlearning on gproblem is sinput variabllearning stepfunctional abpaper. It is othreshold logof switchingmonotonous.monotonous form. If the contains onlcalled isotofunctions). Teach of whimemory elemthreshold funusing more memory elemweights it islearnable for number of problems in

by the smalVΔn of

arger Δmins of the β-c

old zone. g into the nd selecting toise stabilbility of giveds not only othreshold poc relative to his paper oflogical funconly autom

reshold to thes VΔmin when finding

and is deterarator charact amplifiers de thresholdsis determinng learning.

dditional issuproblem is m for a long tcapacitors). Tame idea of output am

he possibilitgiven logicasolved by foles simultane values durin

bilities of LTobvious thatgical functiong functions,

The minfunctions coconcise for

ly positive onous (a The LTE wich containsment, can bnctions. As complicated

ments for kees possible to

an arbitraryvariables.

n experimen

llest changethe β-compV is attaine

comparator cThis is β-comparat

their functionlity and, en logical fuon the minosition of ththe threshol

ffers a methoction. This tematic positihe middle oup to mmaxthe minimu

rmined by thcteristic. Th

with differd hysteresis.nes the valu

ues are addmaintaining ttime (refreshThe solution of applying

mplifiers. Thety of speedal threshold forming weigeously and bng learning. TE is also ext LTE can ins. Accordinall threshold

nimum repoincides witrm of a threvariables, tsubclass of

with the simps only one e taught onlwill be shosynapse cir

eping positivo construct ay threshold fu

Finally, sonts on teach

e of the outparator at d by increas

characteristicachieved

tor two exnal modes.

hence, unctions by

VΔn value e β-comparald of the outod to teach eaching methioning of f VΔmin

VΔmin , whum thresholdhe steepnesse method u

rent threshol The width

ue of Δmin

dressed in the LTE in hing the anaof the problthe thresh

e second isding up Lfunctions. Tghts of seveby changing The problemxamined in tmplement o

ng to the thed functions resentation th their conceshold functthe functionf monotonoplest synopscapacitor asly to isotonoown below,rcuits with tve and negatan LTE thatunction of soome functiohing LTE

tput the

sing c in

by xtra

the the but

ator tput the

hod the but

hich d of s of uses lds, of VΔ

the the

alog lem

hold sue

LTE This eral the

m of this

only ory are of

cise tion n is ous ses, s a ous by

two tive t is

ome onal are

conontea

SPtra7) expHoapp 2

2.1

Thstais t

F

whthr

thaby

can

Rt

F

wh

1 Tthe hav

nsidered andn how to chaching sequeAll results

PICE simulatansistor mod

for analog/periments oorner’s funcplied as test

LTE LeaThreshol

1 Threshold Input W

he conventioarting from ththe threshold

⎜⎜⎝

⎛= ∑

=1wSign

n

j

here jw is thereshold valueRepresentingat a threshold

y the structure

Fig. 1 Gener

It is shownn be represen

1

⎜⎜

⎝

⎛

⎜⎜⎝

⎛=

∑∑

∑

∈

∉

=

xw

xwt

wSign

Sj jj

Sj jj

n

j

here S is a ce

To construct S it iseparating hyper

ving the value 1 o

d a set of rehoose testingences.

in the paption. For the

dels MOSIS /digital circuon LTE teaction of 7 functions.

arnable fold Functi

d Element Weights

nal mathemahe work by Md function:

⎟⎟⎠

⎞− ; SiTxw jj

e weight of te. g a threshold element is e shown in F

ral structure mod

n in [6, 7] thnted in ratio

( ) ; =⎟⎟

⎠

⎞

=⎟⎟⎠

⎞−

A/BRt

STxw jj

ertain subset

is sufficient to takrplane and to inc

on the vertex.

ecommendatg functions a

per were obe simulation BSIM3v3.1 uits were uaching, logic

and 10 va

or Isotonoons

with Contr

atical modelMcCulloch a

⎩⎨⎧

= 1 0

)(Aign

the j-th inpu

d function atraditionally

Fig.1.

of the neurondel.

hat any thresform, as foll

if 1 if 0

⎩⎨⎧

≥<

=

⎜⎜

⎝

⎛

∑∑

∈

∉

BABA

w

wSign

Sj j

Sj j

of indexes1 s

ke any hypercubeclude in S indexe

tions is giveand construc

btained usinexperiment0.8µm (leve

used. In moscal thresholariables wer

ous

rollable

l of a neuronand Pitts [12

≥<

0 if0 if

AA

(1

ut and T is th

as (1) impliey implemente

n threshold

hold functiolows:

(2)

1 =⎟⎟

⎠

⎞−

BB

x

x

jj

jj

such that

e vertex that lies es of the variabl

en ct

ng s, el st ld re

n, ],

)

he

es ed

on

)

in es



∑Cb

doeac

ci

tF

β

Fdw

V

∑ ∈ =Sj j Tw .CMOS implebe like Fig.2.

Fig. 2 β

The voltadetermined bof n- and p-element is caare formedcorrespondin

In [8, 9]controllable introduced:

Rt

SignF

j

n

1j⎜⎜⎝

⎛

⎜⎜⎝

⎛=

∑=

ω

that leads toFig.3a wh

ββ =p .

Fig. imple

In Fig.3b, Fig.3a. The determined bway:

If the outp2/ddout VV ≈ ,

From (2) it ementation o.

β-driven thres

age outV at thby the ratio o-circuits. Foralled β-drivend by cong widths in p], to build input weigh

x

Txw

j

n

jjj

; j

1

=⎟⎟⎠

⎞

⎜⎜⎝

⎛−∑

=

ω

o the β-comhere βnj =

3 The β-commentation; (b

a circuit is output volta

by the value

⎩⎨⎧≤>

=

d

dout V

VV

put voltage o it means t

immediatelyof a threshold

shold elemen

he β-comparof steepnesser this reasonn (β-DTE). T

onnecting tparallel. a threshold

hts, a reduce

Tw

RtT

j

n

j

/

⎜⎜

⎝

⎛=⎟⎟

⎠

⎞ ∑ =

mparator cirβω j ; nβ

mparator: (a) b) equivalen

shown equivage of the β

pn ββα /= i

≥<

if 2/ if 2/αα

dd

dd

VV

of a CMOS that both tra

y follows thatd element can

nt (β-DTE).

rator outputes ( nβ and β

n, the threshThe steepnestransistors

d element wed ratio form

T

xw jj 1 =⎟⎟

⎠

⎞=

rcuit shown

∑ ==

n

j j x1ωβ

,

CMOS nt circuit.

valent to thatβ-comparatorin the follow

11

.

couple (Fig.ansistors are

t n

t is pβ )

hold sses

of

with m is

(3)

in

jx ;

t in r is

wing

3b) e in

nocon

sho

sen

Hereawede

varlowHoantec

a s(Fiof

Sincon

oVsynmo

adouimondothe

2 Fboth

n-saturated ndition thV <

nI

(⎢⎢⎣

⎡−= dpp VI β

In [6] theseown that th

nsitivity αd

dVo

ence, at theaction of theeighted sumcreases as thThe analysiriations madw thresholdsowever, an ad variations chnological) The learnabsufficiently sig.4): the conthe respectiv

nce the synnducting or

out of the βnapses whioment. Clearly, oncding new s

utput state. mplementabilin its base dep

es not depeneir weights (

For simplicity, let h transistors.

mode sincegsout VV −<<

(⎢⎢⎣

⎡−= ddn VVβ

)( −− ddthdd VV

+ pn IIe equations whe suggested

2−≈αout V at

e threshold e β-comparatm Vout 2≈Δ

he threshold iis of β-DTE

de in [5] shows (≤ 3, 4) caartificial neu

of several pcan be comple LTE basesimple contrntrol voltageve synapse.

Fig. 4 β-dr

napse can bnon-conduc

β-comparatorich are con

ce that after synapses do

It followity of β-DTEpends only ond on the num(this fact wa

us assume that th

both of ththV− , gs VV =

2)

2

⎥⎥⎦

⎤− out

outthVVV

() −−− dd

outVV

.0=p were analyzed comparato

the point α

level ( outVtor to a unit

T/2 V, i.e. increases. E stability wed that onlyan be stably uron is a leaparameters (pensated dured on of β-Drol over the e changes the

riven LTE.

be in one octing, the our is formed nducting in

the thresholes not chan

ws from thE and, henceon the threshmber of inpuas established

he threshold volt

hem meet thddV .2 Hence,

,⎥⎥⎦

⎤

,2

)2

⎥⎥⎦

⎤− outV (4

ed and it waor circuit ha

1/ == pn ββα

)2/ddV= thchange of th

it linearl

to parametey β-DTE witimplemented

arnable objec(for exampleing learning.

DTE [8, 9] hainput weigh

e equivalent

of two stateutput voltagonly by th

n this give

ld is reachednge the LThis that the, of the LTold value an

uts and sum od in [6]). Th

age is the same f

he

4)

as as

.1

he he ly

er th d. ct e, . as ht β

s, ge he en

d, E

he E

nd of he

for



ecpvipdt

2Ttoat

g

V

sf

atr

wiw

a

Vαs

V

s

3

m

essential ascomparator topoint. Sincevoltage is resincrease the βpoint is todependency the ratio βα =

2.2 IncreasTo increase ttransistors shoutput voltagamplifier swithe example

Let the gaground and v

ngsV , such t

saturated mofor simplicit

and 0 gsV −<

the currents represented a

nI

ppI −= β

where the paincrease in twhen dsV incr

outV

and

ddV

Let =nλ

2/ddout VV =15.1=α . Par

since the val

5=ddV V an

sensitivity of

3 The values of models.

spect is tho changes in

e the rangestricted withiβ-comparato

o increase of the β-com

pn ββ / .

sing β-compthe sensitivityhould be in thge is in the thitching. Thisof the equivaates of bothvoltage supp

that both th

ode when oVty that p

gsV =

2/ddth VV <−

flowing throas

( gsnn Vβ −=

[)( 2thgsp VV −

+nIarameters nλ

the transistoreases. From

np

p

αλλλα1

+

+−=

(λλ

αnout

dV +

−=

V1 03.0=

, it is easy rameter α dlues of nλ an

nd 15.1=α ,

f the β-com

these parameters

he sensitivitn the current e of β-comin ( ddV−0 ), t

or steepness athe non-lin

mparator out

parator Sey of the β-cohe saturated mhreshold zones can be demalent circuit h transistors ply but by vo

he transisto

2/ddout V= .

gsn

gs VV == ,

2 . Then theough the tra

1()2nth VV λ+

[ (1 ddp V −+ λ

.0=pI

n and pλ reor currents thm these equati

ddVβα

α , =

2)αλλλλλ

np

dpnp V+

++

and 0=pλ

to calculateoes not equand pλ are d

7−=αd

dVout

mparator has

s were taken fro

ty of the at the thresh

mparator outthe only wayat the thresh

nearity of tput voltage

nsitivity omparator, itsmode when te of output onstrated usiin Fig.3b. be fed not

oltages pgsV a

ors are in

Let us assun

thp

th VVV ==

e equations ansistors can

),outV

], )outV

eflect the smhat takes plions we find

pn ββ /

ddV.

V1 11.0 .3 F

e from (6) tal 1 at this podifferent. Wh

5.7 V. Thus,

increased 3

om existing trans

β-hold tput y to hold

the on

s the

ing

by and

the

ume

thV , for be

(5)

mall ace

(6)

(7)

For

that oint hen

the

3.75

sistor

timgre

of by

fedwe

comansynchwe

trarigtheequthesatmochthecurfaccurcurswcurthethisynincthasig

synan

mes. The smeater the senIn the LTE two transist

y the input va

d by the voeight (the cur Let us first mparator whd replace thnapses, baracteristics ere obtained

Fig. 5 Chaequiv

To the lefansistors areght, in the saese characteuivalent trane control voturated modode conditionange. Breakie output signrrents are rect, let the smrrent minI arrent of th

witching of thrrent is not se total currenis synapse. napses also dcreases by aan minI . Thisgnal of the β-The range inapses chanextra n-cha

maller the vasitivity. circuit (Fig.4tors. The gatariable jx ; th

oltage cjV thrrent in the sconsider the

here the synahe couples oy equivalshown in Fiusing SPICE

aracteristics ovalent to the t

ft of the m in the nonaturated moderistics that nsistors are oltage <CVe, if 2>CVn restricts thing this restrnal of the βe-distributed

mallest weighand adding he other syhe LTE. If thsaturated, dent increases, The currentdecrease. Asa value thats leads to a-comparator.in which thege can be e

annel transist

alues of nλ

4), every synte of one trahe gate of th

hat controlsynapse).

e lower part oapse currents

of transistorslent transig.5. These cE simulation.

of the transistransistor cou

mode switchn-saturated mde. It is easy

when ouVin the satura

5.2< V, and 5.2 V. Thus,

he range of coriction leads β-comparator

among the ht corresponthis current

ynapses wilhe synapse wecreasing ouVwill reduce

ts of other s a result, thet is considera decrease i. e control voxtended by tor M3 into

and pλ , th

napse consistansistor is fehe other one i

the variabl

of the LTE βs are summe, which formistors witcharacteristic.

tor that is uple.

ing line, thmode; to thy to see from

5.2=ut V thated mode,

in the nonthe saturate

ontrol voltagto decreasin

r because thsynapses. I

ds to synapst to the totall cause th

with the largesut , because othe current onon-saturatee total currenrably smallein the outpu

oltages of thincorporatinthe circuit a

he

ts ed is

le

β-ed m th cs

he he m he if n-ed ge ng he In se al he st of of ed nt er ut

he ng as



sv

Vrtsstcnst

htFtiirc awto

shown in Figvoltage 1refV

Fi

2/ddout VV ≈reaction of tthe total currsynapse thatsmaller, so transistor opecompensates negative vostabilizes ΣVthrough the s

In Fig.6, whas its maxthrough the Fig.7. It lootwo zones: a is easy to seein the saturreduces, thechange of the When refV

and the linewhich may cthe working of 1refV . In al

Fig. 7 Depe

g.6. The gatesuch that wh

ig. 6 Modifie

, the transisthe voltage rent throught has the smthat gsV b

ens and the for the chan

oltage feedbΣ and therefsynapses. when the conximum valusynapse depks like a trlinear zone

e that when rated mode. e synapse ce synapse cur

1f increases,ear zone of cause the lospoint. Thus,ll experimen

endence of thwhen

e of this tranhen the curre

ed β-compar

stor is satur−=VV refgs 1

h the synapsmallest curr

becomes largincrease in tnge in ΣV . Tback, the efore stabilize

ntrol voltageue ( 5=CV Vpends on oVansistor chaand a zone

5.2≈outV VWhen the

current decrrrent range n, the synapse

the charactss of current , there is an

nts 31 =refV V

he synapse c5=CV V.

nsistor is fedent provides

rator.

rated under

Σ−V . Increases by addin

rent makes ger. The exthe total currThus, due to extra transises the curre

e of the synaV), the curr

outV as shownaracteristic wof saturation, the synapse voltage V

reases and narrows. e current groteristic widestabilizationoptimum va

V.

current on oV

d by

the sing g a

ΣV xtra rent the

stor ents

apse rent n in with n. It e is

1refV the

ows ens, n at alue

out

mopocorthronrefwoof withrmonotwvorefcomis SP

the

Fde

AsvobeshaVo

raton

chadj

ousynvoof

ou

Now let usodified β-coint ( out VV ≈rresponding reshold of re

ne p-channelference voltaorking point.

the charactll be insuffreshold valuodified β-cotion of a ca

wo p-channel ltages 2refVference voltmparator cusaturated a

PICE experimThe depende

e drain of Ml

Fig. 8 Curveependency V

s soon as Mltage gsV of Mcause gsV =

arply grows,)(Iut (Curve

ther good stan the synapse

For compaaracteristics justed to the This experi

utput outV dnapses wholtage cVmina variable. For the old

utput voltage

s consider tomparator (F

2/ddV ), it shto the m

alized functil transistor age refV prov. However, iteristic (Vout

ficient for ge of the curomparator cascode ampltransistors, M

2 and refVtages are s

urrent increasand then Mments 2 =refVence of voltal is shown in

e 1 – depend)(1 IVdM ; Curv

M1 comes intoM2 begins to

13 dMref VV − ., thereby ince 1 in Fig.8)abilization os. arison, Fig.9

of the old function thr

iment showdepends on se control correspond

comparator in the thresh

the p-channeFig.6). At hould provi

maximum vions. To achi

can be usviding its satin this case

)(I at the wgood stabilizrrent. For thicircuit (Fig.ifier [13, p.2M1 and M2, r

3f , respectiselected so ses, first the

M2 becomes 5.3= V, 3refV

age 1dMV on n Fig.8 (Curv

dency )(IVout

ve 3 – depend

o the saturato change at a. The voltagcreasing the). Curve 3 inof the voltag

9 contains and new β

reshold T=89ws how the

the number inputs were

ding to the sm

(Curve 1), thhold point is

el part in ththe workin

de a currenvalue of thieve this goased with thturation at ththe steepnes

working poinzation of this reason, th.6) uses th287] that hareferenced bively. Thesthat as th

transistor Msaturated. I

5.2= V. the current a

ve 2).

; Curve 2 – dency )(IVΣ

tion zone, tha higher speee drop on M steepness o

n Fig.8 showge drop (IVΣ

experimentaβ-comparator9. e comparato

of switchee fed by thmallest weigh

he leap of th32mV. The

he ng nt he al, he he ss nt he he he as by se he Ml In

at

.

he ed M2 of

ws )I

al rs

or ed he ht

he



ntp 2Cddi(smwir

t

tfcttco

Fig. 9 Comold com

new comparathreshold zopoint is abou

2.3 LTE CiCircuits usedetermine thdepend sligimplemented(for examplstructures. Tmemory elemwith floatinginput binaryrepresented.

Fig.10 shothe experime

transistors anform one ocomparator. the lower tranthe gate of thconnection mon the contr

mparator charmparator; Cu

ator has a mone; the voltut 1V.

ircuit and ted to creahe weights oghtly on d. Some of thle, in [14])heir main d

ment they useg gate) and wy variables

ows the coments. Every it

Fig. 10 Th

nd a capacitof the paraThe input va

nsistor and thhe upper one.makes the syrol voltage m

racteristics: Curve 2 for the

much higher stage leap at

the Methodate control of input varthe way

hese circuits ) and theyifference liee (a capacitowith the way

({0,1} or

mplete LTE ts synapse co

e LTE circui

tor. Two ofallel branchariable arrivehe control vo. This order oynapse curremore linearl

Curve 1 for the new one.

steepness in t the thresh

d of Teachinvoltages t

riables of Lsynapses

were publishy have sime in the typeor or a transisy the values

{-1,+1})

circuit usedontains five

it.

f the transisthes of the es at the gateoltage arriveof the transisent dependenly, reducing

he

the hold

ng that

LTE are hed ilar

e of stor s of are

d in

tors β-

e of s at stor ncy by

sevch

synaccalli.ecapappleaonItsam"de

funthrteaas wistotheweouperfir

teawiantauouteaou

mFthr

teaduconrefamsolthea l

simgiv

provarthecom

veral times ange on the cDuring teacnapse currcumulates onlowed to cha. when thepacitor chargproximately arning step. n the accurac value can

mplitude anecrement" siWhen teachnctions (withresholds), theaching algorithe output sth the value

ops. Due to te LTE fireseight changeutput of the βrmissible vaing, by a verTo increase

aching, the Lth different d low. The vught LTE isutput signals aching. Afteutput of the mid will be noreshold voltaThe control

aching, are kue to parasnnection, onfreshing cap

mplifiers witlve this probe control volearning sequThe genera

mulating the ven thresholdThe generaoduces sequeriables xx , 21

e given lombinations.

the influenccontrol voltaching, the vrent (i.e., n a capacitorange only whe input vage increase o

the same The learning

cy required ton be contrnd durationignals. hing LTE fairh a large value learning stithms are ususignal of theof the learn

the small leaafter the v

es its value,β-comparatoralue, which iry small value the marginLTE circuit h

sensitivity value of the s taken from

highF and Fer teaching, β-comparat

ot less than tages of the ot

voltages, whkept on the csitic leakagne should orpacity memth different blem, for exaltages using uence of the fal structuraprocess of

d logical funator of inpences of val

nx,...,2 and thogical functThe teaching

ce of the inage. voltage that

the variar. The capachen the syna

ariable equaor decrease iquanta that g step is appo set the conrolled by cn of "incr

rly complicaue of sum otep should bually built soe LTE begining function

arning step, iariable with, the voltager can exceed is sufficient

ue. n of reliabi

has three outpthresholds: function pro

m the outpulowF are usedthe voltage

or that causthe differencether two amphich have becapacitors ange resistancrganize the

mory. The thresholds a

ample, by authe output sfunction’s vaal scheme teaching the

nction is showput signals lue combinathe sequence tion Y takeg/refresh swi

nput variabl

controls thable weighcitor charge iapse is activeals "1". This realized b

determine pointed basentrol voltagechoosing threment" an

ated thresholf weights an

be small. LTo that as soons to coincidn, the teachinin cases whe

h the smallese leap at ththe minimumfor amplifie

lity after thput amplifierhigh, middl

oduced by thut midF . Thd only durine leap at thses switchine between thplifiers. een set durinnd can changces. In thiprocedure othree outpuallow one tuto-correctinsignal midF aalues.

used whee neuron to wn in Fig.11.

periodicalltions of inpuof values thaes on thesitch passes to

le

he t) is e, he by

a ed s.

he nd

ld nd E

on de ng en st

he m er

he rs le he he ng he ng he

ng ge is of ut to ng as

en a

. ly ut at se o



Fig. 11 General scheme for experiments.

its output F either the signal Y (when teaching) or the output signal midF (when refreshing). The comparator produces the signals "decrement" and "increment." Passive values of these signals are equal to "0" and "1" respectively. Their logical description looks like

Decrement = highFY ⋅ and Increment = lowFY ∨ when teaching;

Decrement = highmid FF ⋅ and Increment =

lowmid FF ∨ when refreshing. Physically, these signals are realized with limited amplitude and duration, determining the learning step.

In experiments with LTE learning, there is an acute problem in selecting threshold functions for teaching, which determines simulation time. The duration of experiments is very important because it is often measured in hours and even days. A threshold function for teaching should have: – a short sequence of variable combinations checking all possible switches of the function value, – a wide range of variable weights, and – a high threshold value for a given number of variables.

This problem will be investigated in greater detail in the final section of the paper. It will be shown that a function that can be represented by the Horner's scheme (...)))(( 4321 −−−− ∨∨ nnnnn xxxxx , satisfies these requirements. For such functions, the sequence of integer values of variable weights and threshold with minimum sum forms the Fibonacci sequence. The length of the checking sequence is n + 1 for the Horner's function of n variables.

2.4 SPICE Simulation Results of LTD Learning Two series of experiments on LTE teaching for given threshold functions are described here.

The goal of the first series was to show the necessity of using a threshold hysteresis when teaching the LTE and when providing the auto-support to the LTE state after the LTE is taught. The

threshold hysteresis can be obtained using three output amplifiers whose characteristics have different thresholds as shown in Fig.12.

Fig. 12 Static characteristics of the output

amplifiers.

When the movement to the threshold is from the left, the higher value of the threshold is used for learning; when from the right, the LTE learns to the lower value. This leads to stretching out the minimum leap outVΔmin of the β-comparator output voltage in the threshold zone and to automatic positioning of the output amplifier threshold with the middle value midF into the middle of this leap. Obviously, the hysteresis width should not exceed

outVΔminmax , which is defined by the parameters of the p-channel part of the β-comparator and by the minimum value minT of the logical function threshold: )/(minmax minminmax TIIfV compout ==Δ where compI is the comparator current in the threshold zone and minmaxI is the maximum current of the synapse with the smallest weight.

For the teaching, Horner's function of seven variables was chosen:

;)))(((

1357235745767

12345677

xxxxxxxxxxxxxxxxxxxxY

∨∨∨=∨∨∨=

.))((

1246346567

12345677

xxxxxxxxxxxxxxxxxY

∨∨∨=∨∨∨=

(8)

From all its possible representations in the form )( 7

17 ∑ =−= j jj TxwSignY with integer values of

weights and threshold, the representation

)21138 532(

76

543217

−++++++=

xxxxxxxSignY

(9)

is the optimum by the criterion of

∑ ==+7

1 54)min( j j Tw . For this representation,

21min =T . The checking sequence for this function contains

eight combinations. Their order is chosen such that the sequence of the corresponding function values



asifaar

rsecmc

fw

lTvc2Op

ttTcmn

w

alternates. Tsequence havis taught tofunction, theat the comamplitude. Trepeated chec

To obtainresults from steepness of experiment, chosen so

ouVΔminmaxconducted wi

The resultfunction of swidths are gi

Fig. 13 Oulearne

(a) wit (b) wit (c) wit

When therlearned the T=267 that stvoltage leapschecking se

704.2732.2 −Obviously, apoor noise-st

In the setaught with athe functionThe disperscomparator min 0 =Δ utVnoise-stabilit

In the thirwith the hy

The combinve one remaro the optimeir change m

mparator outpThe teaching cking sequenn illustrativthe experimthe β-compthe β -comthat when

t was equal ith the step ets of teachinseven variabiven in Fig. 1

utput signal ed to the functhout hystereth hysteresis th hysteresis

re was no hyfunction reptrongly diffes at the comequence fro

)4 up to mafter such tetability. cond case

a 340mV-widn representatsion of th

output deV37.0 ; ma

ty of the LTEd case (Fig.1

ysteresis of

nations in rkable prope

mum represemust cause thtput, which

sequence isnce. e and easi

ment, we hadarator charamparator pa

the threshto 0.5V. Th

equal to 10mng the LTE bles with var13.

of the LTE βction of 7 vaesis, of 340mV, aof 450mV.

ysteresis (Figpresentation ered from the

mparator outpom minΔV

outVΔmax thaaching, the

(Fig.l3b), thde hysteresistion with th

he voltage ecreased c

.0ax =Δ outVE significantl13c), the neu

450mV w

the checkerty: if the Lentation of he voltage le

are equal s a periodica

ily explainad to reduce cteristic. In

arameters whold was he teaching w

mV. to the Hornrious hystere

β-comparatorariables:

and

g.13a), the Lwith thresh

e optimum. Tput vary on

mV28=outVat exceeds 2LTE will ha

he neuron ws and learnedhreshold T=

leaps at considerably

V9. ) and ly increased.uron was tauide. The L

king LTE

the aps

in ally

able the the

were 21,

was

er's esis

r

LTE hold The the (2V. ave

was d to =26.

the (

the

ught LTE

leathethemi

prostathicapcirsevouleamoper"Intimcapper

eas

F

AsoV

comsufresinccapof comno"D

5Cancorouof seq

arned a funce optimum. Ae β-compara

Vout 4.0in =ΔUsing simuloviding autoate based onis aim, the lpacitors werrcuit. The LTven variable

utput amplifiarning modeode. The neriodically rencrement" anme correctinpacitors anrmissible limThe result osily observab

Fig. 14 Corrrefresh

s seen froV485.2=ut c

mbination 1fficient to ssult, the screased by pacitors 7C ,

outV by 33mmbination, 1t sufficien

Decrement" r5 , and 3C cad changing rrection of t

utput signal the function

quence.

ction represeAll the voltaator were ap

V48 ; maxΔlation, we co-support of n of using theakage resise explicitly i

TE was taughes with 450ers thresholde was replaeuron kept sepeated chend "Decremeng the cond supportmits. of correctingble in Fig.14

rection of thehment mode o

m the figcorrespondin

1010110 of switch the osignal "Inc10mV the v

5C , 3C , and mV and switc1010100, thet to switeduced by 1ausing the inthe value o

he control vmidF still co

n on all com

entation that age leaps at pproximately

V55.0=Δ outVchecked the f the LTE inhreshold hysstances of coincorporatedht the Horner0mV wide d. After the aced by thestably functiecking sequeent" occurredontrol voltaing them

g the contro.

e control volof LTE oper

gure, the vng to input variab

output signacrement" ocvoltages on

2C causingching lowF .

e level =outVch highF . 0mV the voncrease of Vof highF . In

voltages, the orresponded

mbinations of

was close tthe output o

y the same V ).

possibility on the learnesteresis. Witontrol voltagd into the LTr's function ohysteresis oteaching, th

e refreshmenioning on thence. Signald from time tages on th

within th

ol voltages i

ltages in the ration.

voltage levethe valu

bles was noal lowF . As ccurred tha

the synapsg the decreas On the nex

V928.2= waThe signa

ltages on 7C

outV by 30mVspite of th

values of thto the value

f the checkin

to of

(

of ed th ge E of of he nt he ls to he he

is

el ue ot a

at se se xt as al

7 ,V he he es ng



tY∨

xY

xT

cdminFc“scst

TFs“

athw dsd0t

In the secotaught to a th

(

7810

91010

xxxxxxY∨∨∨=

∨∨∨=

8910

91010

xxxxxxY

.13579 xxxxx This function

10 = SignY

The checkcontain not defined by thmake the teinterchangabnumber of cFor this pucombination “1”. It is welstar. The toconvenient csequence (ththe checking

x

The checkingFig.15. The signal t that p“decrement”

In the Ladjusted to ththreshold Thysteresis wwas adaptive The learni

duration of signals. Tdischarging 0.15uA and cthe duration o

ond series ofhreshold func

((

56810

678

xxxxxxxx∨

∨∨

∨∨

9679

6789 ((xxxxx

xxx

n can be repr

21132(

87

21

++++xx

xxx

king sequenless than

he terms of eaching seq

ble it is neombinationsurpose, it on which t

ll known thatop vertex candidate to is improves sequence fo

678910 xxxxx1 0 0 0 0 1 1 0 0 0 1 0 0 1 1 1 0 1 1 0 1 0 1 0 0 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 1 1 1 1 1

g sequence ifinal graph

participates isignals.

LTE circuithe maximum

T=89 mmaxidth was 0.8

e. ing step is de

f the “incrThese signa

the capacitcan have a mof the strobe

f experimentction of 10 va

((

346810

345

xxxxxxx∨∨∨

∨∨∨

79457

345 (xxxxxxx

resented in th

553453

9

543

++++

xxxxx

nce for the 11 combina 10Y and Y

quence of fecessary to in the checis possible

the functiont any threshoof the starbe added tthe learning

or the functio 12345 xxxxx

0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 0 1 1 1 0 1 1 0 0 0 1 0 0 1 0 1 0 1 1 0 1 0 1 0 1 1 1 1 1

s implementhic in it repin forming “

t, the β-com sensitivity p

V1min ≈Δ outV85V, and th

efined by theement” andals ensure tors with tmaximum due-signal t (90n

ts, the LTE wariables:

))))

246810

12

xxxxxxxxx =∨

∨=∨

235

12 ))))(xxxxx

(1he form

)898

10

65

−++

xx

. (

function mations that 10Y in (10).

function valhave an o

cking sequene to add an has the vaold function ir is the mo the check

g time). Finaon (10) is

10Y 0 1 0 1 0 1 0 1 0 1 0 1

ted as shownpresents stro“increment” a

omparator wproviding at

V (Fig.9); he learning s

e amplitude ad “decreme

charging athe current uration equalns). It gives

was

;1

910

xxx

=

10)

11)

must are To ues odd nce. any alue is a

most king ally,

n in obe-and

was the the

step

and ent” and

of l to for

F

theis e

cap

ThTh0.2anto of conmoouof if comsigmemuopprofunrefDuou

Fig. 15 Singl

e capacitor oequal to 13.5The change pacitors duri

Fig. 16 Chasynapse

he dynamics he control vo28ms. This md it is possibthe refreshinmode switc

ntrol signal ode if the L

utput F of thsome check

midF and Fmbination o

gnal sets theeans that theust be taugeration can ocess that cnction on sfreshing anduring evaluatutput midF sh

le checking scombin

of 1pF the ma5mV. of the contro

ing the learni

anges of the ce capacitors d

of the learnoltages stop means that thble to switchng mode. Thching can bthat sets the

LTE output e mode swit

king sequenceF do not cf the checkie switcher ine LTE has lght again. T

be interrupcalculates th

some input d evaluationtion, to receivould be gate

sequence of sations.

aximum lear

ol voltages oing is shown

control voltaduring the lea

ing are easilchanging at

he learning ph from the tehe most accue defined we switcher in

midF coincitcher on all e. In the refr

coincide on ing sequencento teachinglost the learnThe refreshipted with ahe value ofcombinationn have to ve correct red.

signal value

rning step tha

on the synapsn in Fig.16.

ages on the arning.

ly observablet the momenrocess is oveeaching modurate momen

with a specianto refreshinides with thcombination

reshing modeat least on

e, this controg mode. Thined state aning mode oan evaluatiof the logican. Obviously

interchangeesults the LT

at

se

e. nt er de nt al

ng he ns e, ne ol is

nd of on al y, e. E



cwraert"v sci

F

VsfFc 2InSlcppf–––

Hfwas

As is easilcontrol voltaweights of representatioare distributeequal to 0.replaced by this moment"Decrement"voltages. The outpu

signal of itschecking seqin Fig. 17. On

Fig. 17 Pictucomparat

outV at the signal midV function. In c

lowF do not control voltag

2.5 ImplemIn order to stnumber of SPICE simlearnable thchoosing testproblem willpaper. As wfunction shou– to have a sh– to cover a w– to have the of variables

MonotonoHorner's schefunctions ofweights and and the lengsequence is

ly seen from ages approxf the varin of the thre

ed close to F28ms, the the refreshm

t, only rare s" appear

ut signals ofs β-compara

quence of thene can see th

ure of the signtor and the L

comparator represents t

cases when tcorrespond

ges are corre

mentability Ltudy the funcexperiments

mulation. Fohreshold eleting thresholl be discussewas already uld match thehort learningwide range o largest thress.

ous Boolean eme match af n variablethreshold for

gth of the sh1+n (the n

Fig.16, stabximately corriables in eshold funct

Fibonacci numteaching mment mode.signals of "Icorrecting

f the LTE ator in one e refreshmenhat the smalle

nals on the oLTE in the re

output is 1the values othe output sito the func

ected.

Limits of thctional powes were carror all expements, theld functions

ed in the finanoticed, a

e following dg sequence, of input weigshold for the

functions reall these demes, the sequrms the Fibohortest learnnumber of c

ble values of respond to the optim

tion (the valmbers). At ti

mode has be Starting frIncrement" asome cont

and the outperiod of

nt mode is givest leap of

outputs of thefresh mode.

V. The outof the realizignals highF action value,

he LTE er of the LTEried out useriments we problem is crucial. T

al section of threshold te

demands:

ghts, given numb

epresentable mands. For suuence of inonacci sequenning (checkicombinations

the the

mum ues ime een rom and trol

tput the ven

e β-

tput zed and the

E, a sing with

of This

the est-

ber

by uch

nput nce ng)

s of

inpwi12

131F

13F

13F

Sinthevoinpseq

(thcom2.7expTadetthevarbe

paradm

mFrannoleacothetecessLTthe

qumaleavarvavo

put variable th three thre:

3213(

87

10

+++=

xxxSign

213(

87

111

+++=

xxxSignF

3213(

87

112

+++=

xxxSign

nce the learnese experimeltages wereputs of the Lquences. In the first

he maximummparator ou7V) was periments a

able 1. Thetermined by e table, the riables is ncause of rela

Table 1:LTE type

10F

11F

12F

In the secorameters omissible thre

mid has been nge borders t less than

arned state. lumn of Tabe amplifierchnological sentially infTE during leae output ampThe other e

uestion: with aintained forarning? Firsriations sholues of the rltage supply

values). Expeshold funct

553432

109

32

++++

xxxx

55342

109

32

++++xx

xx

553432

109

32

++++xx

xx

ning processents, the op

e set on theLTE were fe

series of exm of the utput voltage

determined. are given ine implement

the signal ΔLTE learna

near the edatively small

: Results of S

outVΔ (

1V 0.525V0.325V 1

ond series of the comeshold voltag

defined undthe compar100mV whThe results

ble 1. The cor threshold

parameter fluence LTEarning is adj

plifier from thexperiments what precisi

r normal funst, the LTE sould be invreference voly at ±0.1% (±

periments wetions for =n

);89853 54

−++ xx

1489853

110

54

−+++

xxx

14489853

11

54

++++

xxxx

s was not theptimum value synapses. ed by checki

xperiments, smallest ch

e at the thresThe resu

n the secontability of

outVΔ value. able for fundge of impvalues of Δ

SPICE simul

thVmax)(min÷

1.88÷3.7V 1.9÷3.68V

1.97÷3.65V

of experimenmparator thges of the ouder stipulatiorator produc

hen the LTEs are given onclusion isd (e.g.,

variations)E implemenjusted to anyhese ranges. were associion should th

nctioning of tstability to suestigated. Wltages, when ±5mV), the d

ere conducte10= , 11 an

6 +x

);448 6 +x

(12)

).2334 12

6

−+

xx

e objective oues of contro

The logicaing (learning

ouVΔminmaxhange of βshold level oults of thd column othe LTD iAccording t

nctions of 1plementabilit

outV .

lations.

ddVδ

0.3%0.2%0.1%

nts, for fixehe range output amplifieon that on thced ouVΔminE was in th

in the thir: deviation obecause o

) does nontability. Thy threshold o

ated with thhe voltages bthe LTE afteupply voltag

With constanchanging th

dependence o

ed nd

)

of ol al g)

ut β-of he of is to 2

ty

ed of er he ut he rd of of ot he of

he be er ge nt he of



ttcF2omLc

F

mtp

rcesfv1cmv wsEwtedtmtcwe2

the voltage Vtransistors ofcurrent at ±1

12F , the cumin233I ; 1.5%

of the charaminimum cuLTE will nocurrent chang

Fig. 18 Behathe voltag

On the o

method of prto assume thproportionall

The effectreverse direcchange andexperiments showed that function provoltage chan1. To fix thecondition wamore or less value of not lThe control

with an accshould theyEvidently, thwith the samtotal current either side.determine ththe control vminimum anthe control constant. Thewas the saexperiments.2.

outV from thef the compar1.5% as showurrent in th% of this valacteristic is urrent of thot function pges in this w

aviour of thege ddV change

other hand, roducing refehat the referely to the chant from referenction to thed partially

carried oulearned LT

operly in rege shown in e borders of

as used: signthan the outpless than 50mvoltages of

curacy of 1y be maintahe LTE wil

me threshold of the syn

Experimehe permissiblvoltage CV od maximum voltages of

e condition fame as in The obtaine

e current flowrator shifts alwn in Fig. 1he working lue is mi51.3 I

3.5 times he synapse. properly wheay.

e dependencyes in the inte

taking interence voltagence voltagenges of the vnce voltage c

e effect of y compensaut under thTE 10F , 11Fspective ranthe fourth c

f the rangesal 2/outVΔ sput amplifiermV. f the synapsmV. With ained after ll not functiof the outpu

napses drifts ents were le range ( δ±f one of the currents) ca

f the otherfor fixing the

the previed results are

wing throughlong the axis8. For the Lpoint is ab

in , i.e., the shmore than Evidently,

en the work

y )( pout IV wherval ±0.1%.

o account ges, it is natues must chanvoltage supplchange is in supply voltaates it. T

hese conditio, and 12F c

nges of supolumn of Ta, the followshould be eitr threshold b

es were set what accurathe learnin

ion properlyut amplifier

by 2/minIconducted

CVδ ), in whsynapses (w

an change whr synapses e range bordous series

e given in Ta

h p-s of

LTE bout hift the the

king

hen

the ural nge ly. the age The ons can

pply able

wing ther by a

up acy ng? y if the on to

hich with hile are

ders of

able

T

F

FF

I

ranthichcormi

baof simacc

3.

A an be exbyinpknwereshavtypincbuthi 3.1ThthrLTfun

wh

dofeeinvressyn

Table 2:Type

minSIδ

10F m42.0 I±

11F m40.0 I±

12F m34.0 I±

In the secondnges of synaird and fouange of the rresponding inimum and mIt is possiblsed on Table

synapses imultaneouslycurate as uni

LTE LeaThresho

threshold fuisotonous Brealized by

citatory inpuy artificial neputs. If the i

nown beforeheight sign ispective varve synapsespe of the increment and

uilding such sis section.

1 Statemenhe behavioureshold funcTE, it is nctions in red

∑=

=n

j

wRtF1

(

here w jj =ω

The simplesubling the neding the Lversions jxspectively. Nnapses does

Results of Sn CVδ

min ( %3.5 ±±

min ( %7.4 ±±

min ( %8.3 ±±

d column of apse current

urth columncontrol voltchanges of

maximum wle to make e 2 data: sincin the LTEy, their maits of millivo

arnable foold Functiunction with Boolean func

an artificialuts. Howeveeural networnput type (ehand, the pis solved triable. Other capable of

nput during decrement s

synapses for

nt of the Prur of a β-Dction in ratioconvenient duced ratio f

=jj RTxw )/

Tj / and Rt

st way of sonumber of

LTE inputs with inpu

Note that dnot lead to

PICE simulamin 46mV)± 0±40mV)± 0±32mV)± 0±

f the table, tht change are

ns contain ttages. These current in s

weights. the followince all the con

E should beaintenance solts.

or Arbitraions positive inp

ction. Such al neuron (LTer, most prorks also requexcitatory or problem of rivially by rwise, the nf forming th

the learningsignals. The r the LTE is

roblem DTE is deso form [6].

to represeform:

∑=

n

jjj xRt

1

)( ω

⎩⎨⎧

= if 1 if 0

)(AA

A

olving this pvariables (aby both x

ut weights doubling thecutting dow

ations. maxCVδ

17mV)( %60. ±

27mV)( %73. ±

10mV)( %23. ±

he permissible shown. Ththe limits o

limits definsynapses wit

ng conclusiontrol voltagee maintainehould be a

ary

put weights ia function caTE) with onloblems solveuire inhibitor

inhibitory) iinverting thinverting th

neuron shoule weight ang, using onlpossibility othe subject o

scribed by To build th

ent threshol

(13

≥<

.1,1

AA

problem is band synapses

jx and the

ja and be number o

wn the numbe

le he of ne th

on es ed as

is an ly ed ry is

he he ld nd ly of of

a he ld

3)

by s) ir

jb of er



of realizable threshold functions because the implementability of LTE depends only on the threshold value and does not depend on the sum of the input weights or number of synapses. On the contrary, incorporating extra inverse inputs increases the number of realizable threshold functions of n variables by 2n times.

Let in a certain isotonous threshold function

∑ =

n

j jj xRt1

)( ω some variables Yxi ∈ be inverted

while other variables ),( XYZjiZx j =≠∈ U are not. Then

⎟⎟

⎠

⎞

⎜⎜

⎝

⎛

−

−

=−−−

=−−+

=−+=

∑∑ ∑

∑ ∑ ∑

∑ ∑

∑ ∑

∈

∈ ∈

∈ ∈ ∈

∈ ∈

∈ ∈

Yx i

Zx Yx iijj

Zx Yx Yxiiijj

Zx Yxiijj

Zx Yxiijj

i

j i

j i i

j i

j i

xxRt

wTxwxwSign

TxwxwSign

TxwxwSignF

ω

ωω

1

))((

))1((

)(

(14)

where Tw jj /=ω . It is easy to see from (14) that the use of negative weights can be reduced to inverting the variables and vice versa. The normalized threshold of a function represented by Rt-formula with negative weights is equal to ∑ ∈

−Yx iiω1 .

The circuit of a neuron synapse capable of forming both positive and negative weights of an input variable is made of two simple synapses as shown in Fig.19.

Fig. 19 Synapse forming positive and negative

weights of the input variable.

It is easy to see that the LTE with such synapses realizes the threshold function

⎟⎟

⎠

⎞

⎜⎜

⎝

⎛

−

−

∑∑

=

=n

j j

n

j jjj

b

xbaRt

1

1

1

)( (15)

where ja and jb are weights brought to the threshold. They are defined by voltages on the capacitors 1C and 2C for jx and jx , respectively. On the other hand, for the case of doubling the synapses number, it follows from (14) that the threshold function realized by the LTE must be

⎟⎟

⎠

⎞

⎜⎜

⎝

⎛

−−−

−

∑∑

=

=nj jjjj

nj jjj

abSignab

xbaRt

1

1

)()(1

)( (16)

(to maintain the limitations on weights and thresholds).

It is easy to see that if in every pair ),( jj ba one of the weights is equal to zero, then expressions (15) and (16) coincide and have the form:

.1

)(

⎟⎟

⎠

⎞

⎜⎜

⎝

⎛

−

−

=+=

∑∑ ∑

∑ ∑

∈

∈ ∈

∈ ∈

Yx i

Zx Yx iijj

Zx Yxiijj

i

j i

j i

b

xbxaRt

xbxaRtF

(17)

It follows from the above that when teaching an LTE with such synapses it is desirable to change the input weights ),( jj ba in such a way that one of the weights in each pair goes to zero. Moreover, as it will be shown below, this condition provides the maximum level of LTE implementability.

It is difficult to conclude from (15) and (16) that synaptic weights affect the neuron implementability. Let us look at how the β-comparator operates (Fig.6). The sizes of p-transistors and reference voltages 2refV and 3refV determine the current thI when the output voltage of the β-comparator is equal to the output amplifier threshold. As a first approximation, the smallest change of the current is

TII th /0 = and 0kIVout =Δ where k is the steepness of the β-comparator voltage-current characteristic at the threshold of the output amplifier. However, if

0≠ja and 0≠jb , then via each j-th synapse an additional current flows that is determined by

),min( jj ba . Thus, the approximate value of the smallest current can be obtained from the equation

∑−=j

jjth baI

TII ),min(00

and

∑+=

j jj

th

baTII

)),min(1(0 . (18)

It follows from (18) that if the value of outVΔ is fixed, the largest realizable threshold depends on

),min( jj ba as

∑+Δ≤

j jjout

th

baVkIT

)),min(1(. (19)



eapta

hethndnlim

3TFttpcvcdfrbviLastr

osls

Thus, keepeither increaactually assproviding mthe synapsealgorithm.

In [20] sehave been sexistence of to keep realizhas been pronot find on-cdecisions. Onext subsectilearning prindependent modified syn

3.2 The ProThe same geFig.11, is uteaching thethreshold funperiodically combinationsvalues that thcombinationsdepending ofunction. Therepresented ibe derived fvariables witis supposed LTE. Other assumed to schematics transistors arreplicable.

The “Toutput F eithsignal midF logical funschematic is

Fig

ping the impasing thI dusociated wi

),min( =jj ba circuit an

veral modifisuggested astable decisizing non-isooved. Unfortchip learning One possible ion. This sorocess, wh

of initial napse circuit.

oblem Decieneral structuused when e LTE annction. The produces ch

s of input vahe given logs. The isotoon 10 variae checking sin Fig.15. Nfrom this futh the help oto be implem

blocks of be imple

(shown belre pointed ou

Teach/Refreshher the signa

(when refnction YF &=very simple

g. 20 Schema

plementabilityuring the leith some d0 for any j

nd changing

fications of snd for eachions, which totonous threstunately, the algorithms lsolution is p

olution provihich gives

conditions

ision ure scheme tsimulating t

n arbitrary “Input Sig

hecking sequariables and tical function

onous logicaables is choequence for

Non-isotonouunction by of inverters. mented sepa

f the generaemented onlow), the ut to make t

h Switch” al Y (when tfreshing) an

FTeach mid& ∨and is realiz

atic of the sw

y level requiearning (thatdifficulties) j by modifyg the learn

synapse circuh of them the LTE is ashold functioe authors coleading to thproposed in ides an on-c

convergenand uses

hat is shownthe process given logi

gnal Generatuence of vathe sequencen takes on thl function (osen as a tthis functions functions cinverting so The genera

arately from al scheme

n-chip. In widths of

the experime

passes to teaching) or nd realizes

Teachd & .zed in Fig.20

witch.

ires t is

or ying ning

uits the

able ons, ould hese

the chip nce a

n in of

ical tor” alue e of hese 10) test n is can

ome ator the are the all

ents

its the the Its

.

“InFig

deconwi

whsig“LreaM8is com(trtra“Incontrap-cUswi

dede

thatra“Limtha(trtraof

Fig. 21 S

The “Compncrement” sg. 21. In the schsignated as ntrols p-chanth its implem

incr _here F is thgnal inversioLog.1” and acalized on the8. The outputa NAND gambined wiansistors M

ansistors Mncrement” cntrolled by t

ansistor M4 pchannel transing the voldth of the tra“Decrement

necr _ , ndecnecr _ has th

decr _at is implemansistors M9Log.1”. Themplementationat containsansistors M1

ansistors M1the signal d

Schematic of

parator” prosignals. Its

hematic, thepincr _ to

nnel transistmentation des

FFp low ∨=_he switch ouon. Passive ctive value ise gates G1, t stage of theate (transistoth an emb1– M5) and

M4 and M5current throuthe signal inprovides the nsistor of theltage divideransistor M4 mt” signals c

ncr _ , and nhe logical fun

high FFn ⋅=_mented on th

– M14. Actie last stn is an inve the emb

10, M11) and 2, M14, whic

ndecr _ . Th

f the “Comp

oduces “Decschematic

e “Incremenpoint out thtors. Its funcscription is

FFtF low ⋅=∨utput and t

value of ts “Log.0”. TG2, and tran

e function imors M2, M3, bedded vola current m

5, which ugh p-chann

pncr _ . The0.15uA curr

e minimum r allowed tmore than twconsist of t

nnincr _ . Thnction

FFttF ⋅∨=⋅he gates G3,ive value of age of t

erter (transistbedded volt

the current ch restricts t

he transistor

arator”.

crement” anis shown i

nt” signal ihat this signaction togethe

tF ∨ is the strobe

this signal ihe function insistors M1

mplementatioM7, and M8

ltage dividemirror on threstricts th

nel transistore width of thrent through width (1.2uo reduce th

wenty times.three signalshe main signa

highF , G4, G6, anthis signal i

the functiotors M9, M13tage dividemirror on ththe amplitudM14 of 11.7

nd in

is al er

e-is is –

on 8) er he he rs he

a u). he

s: al

nd is

on 3) er he de 7u



width provides the current 0.15uA through n-channel transistors of minimum width (1.2u) controlled by the signal ndecr _ .

Two additional “Decrement” signals ( nnincr _ and nndecr _ ) are used when teaching the LTE synapses input weight signs. Each of them creates an additional force that pulls down voltages of the synapse capacitors corresponding to ),min( jj ba up to the ground potential during LTE learning. The signal nnincr _ is alternative to the signal pincr _and has the logical function

tFFtFFnnincr lowlow ∨⋅=⋅⋅=_ that is implemented on gates G2, G5, and transistors M15 – M20. The last stage of the implementation is analogous to the last stage of the signal ndecr _ . The transistor M20 of 21.5u width provides current 81.6nA through n-channel transistors of minimum width (1.2u) controlled by the signal nnincr _ . The signal nndecr _ is alternative to the signal

ndecr _ and has one additional restriction: if during the strobe-signal "1Log."=t the signal =ndecr _ ”Log.1” is finished, the signal =nndecr _ “Log.1” cannot be produced. The function of the signal

nndecr _ is defined as Q_ ⋅⋅⋅= tFFnndecr high

where Q is output of the latch keeping the value of the signal ndecr _ up to the end of the strobe signal. In Fig.21 the latch Q is constructed from gates G8, G9 and has excitation functions

. ; FtRFtFS high ∨=∨⋅= The signal nndecr _ is implemented as

Q_ ⋅⋅∨⋅∨= highFFtFtnndecr on gates G3, G4, G6, G7, and transistors M21 – M26. The last stage of the implementation is the same as in the realisation of the signal ndecr _ . The transistor M26 of the current mirror with 11u width provides current 0.16uA through n-channel transistors of minimum width (1.2u) controlled by the signal nndecr _ .

The LTE itself consists of β-comparator with output amplifiers and synapses. The schematic of the β-comparator is presented in Fig.22. In this figure, all parameters of transistors and values of reference voltages are pointed out. All amplifiers are constructed as a serial connection of three inverters. The amplifier with the output midF has a threshold equal to 2.7V. The thresholds of

Fig.22 Schematic of the β-comparator with output

amplifiers.

amplifiers with outputs lowF and highF are equal to 2.3V and 3.15V respectively. Thus, the width of threshold hysteresis is 850mV. The full synapse circuit of the LTE learnable for arbitrary threshold functions is introduced in Fig.23. It is constructed on the basis of the synapse circuit in Fig.19. Voltages 1CV and 2CV on the capacitors control the synapse currents flowing through pairs of transistors (M5, M21) or (M6, M22). The circuit has two input logical variables x and x . The variable x can be derived with the help of an inverter. The voltages on the capacitors C1 and C2 correspond to the positive “a” and negative “b” weights respectively. If thC VV <1 or thC VV <2 (here thV is the threshold voltage of n-channel transistors), it means that 0=a or 0=b because the corresponding transistor pair will be closed.

The signals pincr _ and ndecr _ increase and decrease the capacitor voltages through pairs of transistors (M1, M3), (M2, M4) and (M7, M23), (M8, M24) respectively depending on the value of input variables (x, x ). Two pseudo n-MOS inverters on transistor pairs (M29, M39) and (M30, M40) are sensitive elements of capacitor voltages close to thV . Voltage =4refV 3.9V fixes the conductivity of their p-channel transistors. Output signals G1 and G2 of these elements control the conductivity of two pairs of transistors (M15, M17) and (M16, M18) respectively. Transistors of each pare can pass currents only when the voltage on the corresponding capacitor (

1CV or 2CV ) exceeds 675mV. Two inverters (M31, M41 and M32, M42) with outputs G3 and G4 invert the signals G1 and G2. Signals G3 and G4 control



Fig. 23 The LTE Synapse implementation.

transistors M9 and M10, each of which opens when the voltage of the corresponding capacitor ( 1CV or

2CV ) exceeds 635mV. Difference between these threshold voltages (675–635=40mV) is very important because it provide correct learning behaviour of synaptic weights in the region close to the threshold of n-channel transistors. Signals G3 and G4 also write information about the sign of the synaptic weight in the latch (outputs Q and Q ) on transistors M33 – M38, M43, M44. The latch keeps information when voltages of both signals G3 and G4 exceed the threshold of the latch inputs. When Q=Log.1, the weight sign is positive. If Q = Log.0, the sign is negative. Signals G1 – G4, nndecr _ , nnincr _ , the latch Q, and input variables ( xx , ) control the conductivity of additional decrement chains for each capacitor. For the capacitor C1, these chains are described by the expression

)MMM(MMMMMM 26172719132315119 ∨∨ . For the capacitor C2 the expression for chains is

)MMM(MMMMMM 251828201424161210 ∨∨ . It should be noted that transistors M27, M28 cannot be replaced by transistors M23, M24 because of

appearance of parasitic dependency between voltages 1CV and 2CV through parasitic capacitances of these transistors.

The initial setting signal is controls the transistors M45 and M46, which serve only for the initial setting of the voltages on the capacitors C1 and C2.

During learning, the synapse works in the following way. Let us suppose that initially

thC VV <1 and thC VV >2 . Then the signals G1 and G2 will be equal to “Log.1” and “Log.0” respectively and the latch Q will be in the state Q = Log.0 (negative weight sign). There are two cases.

First, the sign of the input variable weight is negative, i.e., it coincides with the state of the latch. In this case the signals nnincr _ and nndecr _ pull together the voltage 1CV to 0V by the chain (M9, M11, M15, M23) and by the chain (M13, M17, M26) respectively. At the same time the signals

pincr _ and ndecr _ try to set the voltage 2CV corresponding to the weight of the input variable x . Second, the sign of the input variable weight is positive, i.e., it does not coincide with the state of the latch. In this case, the learning sequence will



provide that for the capacitor C1 increment steps caused by the signals pincr _ will prevail over decrement steps caused by the signals ndecr _ ,

nndecr _ , nnicr _ and the voltage 1CV will grow. As soon, as 1CV exceeds thV the sensitive inverter (M29, M39) closes the transistors M15, M17 halting the action of the signals nndecr _ , nnicr _ by the additional chains. This inverter also switches the inverter (M31, M41), which, in its turn, opens the transistors M10 and M28 enabling action of the signals nndecr _ by the additional chain. After that, the voltage 1CV continues to rise to the weight value and the signals ndecr _ , nndecr _ will pull down the voltage 2CV by two chains: (M8, M24), and (M14, M20, M28).

As soon as the voltage 2CV reaches 675mV, the sensitive inverter (M30, M40) opens the transistors M16, M18 and, when mVVC 6352 = , switches the inverter (M32, M42), which, in its turn, closes the transistors M9 , M27 and switches the latch Q into the state Q = Log.1 (positive weight sign). Output signals of the latch close the transistor M20 and open the transistor M19. The difference in critical values of the voltage 2CV , which leads to switching the sensitive element (M30, M40) and the inverter (M32, M42), is very important because in this case switching of the latch only slightly changes the condition of the capacitor C2 discharging and the voltage 2CV continues decreasing down to ground potential due to opening the additional chain (M14, M18, M25) that partly compensates closing the chain (M14, M20, M28).

3.3 Results of SPICE Simulation All experiments on teaching of the LTE to non-isotonous (antitonous) threshold functions were conducted for functions obtained by inverting some variables in the isotonous threshold Horner's function (10) of 10 variables. Below the results of SPICE simulating LTE learning are presented for only two test-functions: for the isotonous function (10) and for the non-isotonous function (antitonous) derived from (10) by inverting variables with even indexes

; 1246810346810

56810781091010


∨∨∨∨=

. 1357923579

4579679891010


∨∨∨∨∨=

(20)

Another form of the function (20) representation is

).34553421138532(

109876

5432110

−+−+−+−+−+−=

xxxxxxxxxxSignY

(21)

The checking sequence for these two functions is represented in Fig.15. In experiments, the duration of keeping one combination is 200ns and the checking sequence takes 2.4us. The learning sequence is cyclically repeated checking sequence.

Fig.24 shows the LTE learning process for the function (10) starting from the initial state, in which

V021 == CС VV for all synapses. In this figure, designations jVΔ of curves denote voltage difference 21 CC VV − for the synapse of j-th variable.

Fig. 24 The LTE learning of the function (10).

It is easy to see from Fig.24 that all curves reach stable states for the time equal to 0.44ms or for 183 learning cycles and the weights of all variables are positive.

Fig.25 illustrates the process of LTE learning for the function (20) starting from the same initial state.

Fig. 25 The LTE learning of the function (20).

It is possible to conclude, analyzing Fig.25, that this learning process has the same time parameters as the process in Fig.24. Signs of variable weights and their values are determined correctly: all even variables have positive weights and all weights of odd variables are negative.



In Fig.26 the process of LTE learning of the function (10) is presented for one of the worst cases when from the initial state, in which all synaptic weights are the least negative ( V01 =CV ,

V52 =CV ), the LTE is taught to all positive weights.

Fig.26 The LTE learning of the function (10) from

the initial state of the least negative weights. The picture shows the correct result of the learning. The decision is found for 1.2ms (500 cycles).

Fig.27 illustrates the process of LTE learning of the function (20) from the initial state ( V01 =CV ,

V52 =CV ) for all synapses. Analyzing Fig.25, it is possible to conclude,

that this learning process is finished for 1.45ms (604 cycles). Signs of variable weights and their values are determined correctly: all odd variables have negative weights and all weights of even variables are positive.

Fig.27 The LTE learning of the function (20) from

the initial state of the least negative weights.

For the proposed procedure of on-chip LTE learning for arbitrary threshold functions of some number of variables, it is not clear how to prove its convergence analytically. Sufficient SPICE-simulation experiments have been done to establish

that this learning procedure possesses of convergence from initial states ( V01 =CV ,

V02 =CV ), ( V01 =CV , V52 =CV ), ( V51 =CV , V02 =CV ),( V51 =CV , V52 =CV ), which may be

different for different synapses. To provide the procedure convergence from these initial states it is necessary to chose very accurate current amplitudes of the signals nndecr _ and nnicr _ .

Nevertheless it is impossible to be sure that there is no an initial state, from which does not exist the algorithm convergence. By this reason it does not possible to confirm that, if the LTE is taught to realize some threshold function, using its state as initial, it can be repeatedly taught any other function. May be it is so, but this fact did not proved.

In any case, if accept the restriction that any learning process is started from the reasonable initial state ( V01 =CV , V02 =CV ), which is the same for all synapses, the accuracy diapason of setting for the signals nndecr _ , nnicr _ become much wider.

By this reason, it can be recommended after losing by the LTE of a learned state to do the initial setting of the LTE before new act of learning.

4. Some Functional Problems in Experiments with LTE Learning Developing a hardware implementation (e.g. CMOS) of artificial neurons with critical parameters such as threshold value of realizable functions, number of inputs, values and sum of input weights is a difficult technical problem that nevertheless has to be solved [1-8]. While tasks of purely logical design can be solved more or less efficiently in an analytic way, tasks of physical design necessarily require computer simulation (e.g., SPICE simulation). Computer simulation can and must answer the following questions: − What are the limiting parameters of an artificial neuron of a certain type? − Are these parameters attainable during teaching?

A modern learnable artificial neuron, which is implemented as a hardware device and oriented to reproducing complicated threshold functions (sum of input weights and threshold >1000), is merely a sophisticated analog-digital device, whose maximum functional power is attained, in many cases, by using the effects of the second and even the third order transistor behaviour. This, in its turn, requires using models of higher levels (e.g. BSIM3v3.1). Therefore, the dependencies of



neuron behaviour on the synapse parameters are, generally speaking, non-linear. Because of this, the neuron for simulation should have a wide range of synaptic weights that covers all the range of values under consideration. This, in turn, requires that the neuron behaviour should be simulated under all necessary combinations of the input signals. It should also be taken into account that simulation results strongly (and sometimes crucially) depend on the parameters of the transistor model in usage. Hence, to get results with a certain level of generality, a number of simulations using different models (e.g., from different manufacturers) should be conducted. Thus, to get an answer to the question about the maximum functional power of the artificial neuron, serious experimental work is needed, the volume of which obviously depends linearly on the number of variable value combinations used in every experiment.

The existence of a number of neuron circuit parameters, which provide reproducing a threshold function of limiting complexity, does not mean that the voltages controlling the input weights and threshold for this function can be attained during the teaching. If a control voltage V can change in the interval maxmin VVV ≤≤ and maxw is the maximum value of the corresponding input weight (or threshold), then maxminmax /)( wVVV −=Δ is the value determining the required precision of teaching. Taking into account the possible non-linearity of the dependence )(Vw and necessity of compensating during teaching technological variations of transistor parameters (geometrical sizes, thresholds, etc.), the precision of setting the control voltages should be VkV Δ=δ (k < 1). A teaching system can be considered as a complex non-linear analog-discrete control system with feedback delay. Its analytic study is very difficult, so again one arrives at the necessity of computer simulation as the basic tool of this research.

In order to provide the required precision, the increment of the voltage controlling the synaptic weight in one step of teaching (exposing one combination of variable values) should be Vδ≤ . Hence, to simulate the process of teaching the artificial neuron to reproduce a threshold function with 200100max ÷=w , every combination from the learning sequence should be exposed about 1000 or more times, regardless of selecting the teaching strategy. Because of this, SPICE simulation of the teaching process takes hours. Naturally, the duration of the simulation process linearly depends on the length of the learning sequence.

In order to obtain reasonable simulation time for highly complicated artificial neurons, the test tasks should be threshold functions with learning sequences of the minimum length and with fixed values of maxw . Values of input weights should cover all the value ranges as tightly as possible. Functions of this type are the subject of the section. Some results of threshold logic that have been known for several tens of years, at least as scientific folklore, will be used. 4.1 Bearing Sets of Threshold Functions and Checking Sequences The geometrical model of threshold functions

∑ =−=

n

j jj xwSignY1

)( η is a separating hyperplane

with the equation∑ ==−

n

j jj xw1

0η . If the

combinations of variable values correspond to vertexes of a unit hypercube, the threshold function has the value "Log.1" on the vertices that have a positive distance from the separating hyperplane (T set) and the value "Log.0" on those whose distance from the separating hyperplane is negative (F set). Traditionally, the task of synthesizing a threshold element by given sets T and F is reduced to solving a linear programming task:

finding ⎟⎟⎠

⎞⎜⎜⎝

⎛+∑

=

n

jjw

1

min η at the conditions

⎪⎩

⎪⎨

⎧

<−

≥−

∑

∑

∈

∈

Fxjj

Txjj

j

j

xw

xw

0

0

η

η

.

Note that the system of inequalities is redundant since some inequalities majorize others.

Without loss of generality, threshold functions with 0>jw will be only discussed further. Such threshold functions correspond to isotonous Boolean functions (monotonous functions with only positive variables) and can be realized by neurons with only excitatory inputs. By definition, for monotonous functions )(Xf , from ij XX > it follows that )()( ij XfXf ≥ where jX and iX are certain combinations of variable values. Or, for isotonous threshold functions

∑∑∈=∈=

−>−ikjk Xx

kkXx

kk xwxw11

ηη .

A monotonous Boolean function in a unit hypercube corresponds to a "star", i.e. a set of subcubes that have at least one common vertex (star vertex) [23]. For an isotonous function, the star vertex is }1,...,1,1{max =X ; for an antitonous



function (inversion of the isotonous one) –}0,...,0,0{min =X . The vertex lying on the maximum

diagonal of a subcube (at the maximum distance) from the top of the star will be referred to as a bearing vertex. The set of bearing vertices for star T will be called bearing set 0T and for star F – bearing set 0F . It is easy to see that vertices 0TX ∈ are the minimum and vertexes 0FX ∈ are the maximum in the respective subcubes. Hence, to solve the linear programming task, it is enough to use only the inequalities corresponding to the bearing sets.4

A subcube of dimension m in an n-dimensional hypercube corresponds to a conjunction of range

mn − in the concise form of a Boolean function.5 This conjunction determines the bearing vertex, namely: for the set 0T , coordinate jx has the value "Log.l", if jx appears in the conjunction and "Log.0" otherwise; for the set 0F , coordinate jx has the value "Log.0", if jx appears in the conjunction and "Log.l" otherwise. For example, for 7=n ,

1010010631 ⇔xxx , 01001016431 ⇔xxxx . Thus, the number of vertices in the sets 0T and 0F is equal to the number of terms in the minimum Boolean forms of the threshold function and its inversion.

It obviously follows from above that, if the artificial neuron is taught to recognize bearing sets, it recognizes corresponding threshold function as well. A learning sequence that consists of input variable value combinations belonging to bearing sets will be referred to as a bearing learning sequence. The length of the bearing learning sequence can vary in a wide range: from 1+n for the function )(

1∑ =−=

n

j j nxSignY up to

)!2/()!2/(!22 2/

nnnC n

n⋅

= for the function

)2/(1∑ =

−=n

j j nxSignY with odd n.

4.2 Test Functions The length of the test sequence as a function of the number of variables means nothing, if it is not correlated with complexity of the threshold function. A natural question arises about estimating threshold function complexity. For simulation

4 This result has been known in the threshold logics since the late 50's or early 60's, but it is difficult to give specific reference. 5 For a monotonous function, the concise form coincides with the minimum form.

tasks, which are discussed here, this complexity is associated with implementability of an artificial neuron. It varies depending on a circuit solution. For a ν-CMOS artificial neuron [3-5], its implementability and, hence, threshold function complexity estimation is determined by the sum of the input weights. For a β-driven artificial neuron [6-8], implementability and complexity are determined only by the threshold value. Keeping in mind these two types of complexity estimation for threshold functions, we will use two efficiency criteria for test functions, namely: η/)(1 nLC = and

∑ ==

n

j jwnLC12 /)( where )(nL is the length of the

learning sequence in the number of bearing combinations. The lower the values of these criteria, the more efficient the function will be in teaching.

Let us start from a simple example considering two threshold functions:

j

n

jj

n

j

n

njj xxnxsignY V

11

111 Y ; Y;

===

==⎟⎟⎠

⎞⎜⎜⎝

⎛−= Λ∑ and

; )1(1

12 ⎟⎟

⎠

⎞⎜⎜⎝

⎛−+−= ∑

−

=

n

jjn nxxnSignY

. Y ;1

12

1

1j2 V j

n

jnj

n

n xxxxY Λ−

=

−

=

∨=⋅=

Both functions have the same number of bearing combinations and 1)( += nnL . Since the threshold is the same for 1Y and 2Y , both functions also have the same value of the first efficiency criteria 1C ,

nYCYC /11)()( 2111 +== . The only advantage of

2Y is that there is an input with the weight 1−n . At the same time nYC /11)( 12 += ,

)1/(1)2/1()( 22 −+= nYC . Since )()( 2212 YCYC > , 2Y is preferable as a test function; the question

arises of whether it is possible to derive test functions with the highest efficiency for both criteria.

Let us consider Boolean functions that can be represented in Horner's scheme:

...))(()( 43211 ∨∨∨= −−−− nnnnn xxxxxnH , (...)))(()( 43212 −−−− ∨∨= nnnnn xxxxxnH ,

and call them Horner's functions of the first and second type, respectively. Note that according to De Morgan's law, inverted functions of the first type are functions of the second type with respect to inverted variables, and vice versa, i.e., inverted functions of the second type are functions of the first type with respect to inverted variables.



Let )]([)]([ 0201 nFNnTN = , )]([)]([ 0102 nFNnTN =

be the numbers of vertices in the bearing sets of Horner's functions of n variables of the first and the second type. It is easy to see that

)1()( 21 −∨= nHxnH n and )1()( 12 −= nHxnH n . Therefore,

1)]1([)]([ 0101 +−= nFNnTN , )]1([)]([ 0101 −= nTNnFN and

11)1()( +=+−= nnLnL . Hence, a Horner's function of n variables has the shortest bearing learning sequence.

Statement 1: The first and second types of Horner's functions of n variables are threshold functions with the same vectors of input weights

},...,,{ 11 wwwW nn −= differing only in their threshold values.

It is easy to find, directly applying De Morgan’s theorem, that Horner's functions of the first and the second type are dual6, i.e. )()( 21 nHnH d= .

For any threshold function

∑ =−=

n

j jj xwSignXf1

)()( η ,

the following is true:

⎟⎟⎠

⎞⎜⎜⎝

⎛−+−= ∑

=

n

jjj xwSignXf

1

1)( η

and

=⎟⎟⎠

⎞⎜⎜⎝

⎛−+−−== ∑

=

n

jjj

d xwSignXfXf1

1)1()()( η

⎥⎦

⎤⎢⎣

⎡⎟⎟⎠

⎞⎜⎜⎝

⎛+−−∑ ∑

= =

n

j

n

jjjj wxwSign

1 1

1η

that proves the Statement 1. Statement 2: Input weights of a threshold

functions represented by Horner's scheme form the Fibonacci sequence.

It follows directly from the minimum form for Horner's functions that

2121 )1()( −− +=−== nnn wwnnw ηη where )(1 nη and )(2 nη are thresholds for Horner's functions of n variables of the first and the second type, respectively. Solving the difference equation with the initial conditions 121 == ww , we get

⎥⎥⎦

⎤

⎢⎢⎣

⎡⎟⎟⎠

⎞⎜⎜⎝

⎛ −−⎟⎟

⎠

⎞⎜⎜⎝

⎛ +=

++ 11

251

251

51

nn

nw ;

6 Functions )( Xf and )( Xϕ are called dual, if

)()( XXf ϕ= .

∑=

++ −===n

jnjnn wwwnwn

11121 1 ;)( ;)( ηη .

At first glance, Horner's functions look like functions of n variables with extreme parameters (sum of input weights, threshold, etc.). However, this is not so, as it is possible to see from simple examples. Already for fore variables there is a threshold function

)5322()(

4321

432321

−+++=∨∨

xxxxSignxxxxxx

(22)

with the sum of input weights greater than that of Horner's functions. From the function dual to (22), by deleting inversions of variables and multiplying it by 5x the next function is derived

).95322())((

54321

5432132

−++++=∨∨∨

xxxxxSignxxxxxxx

(23)

This function has a threshold greater than that of the second type Horner's function of five variables. However, both functions (22) and (23) have bearing learning sequences of the length equal to n+3. Note that for values within practical interest of maximum input weights, thresholds, and sums of input weights (100-1000), Horner's functions are an excellent example of test functions.

Finally, the following Horner’s functions can be recommended as test functions with the shortest learning sequences:

);2121 138532(

8

76543218

−+++++++=

xxxxxxxxSignY

);343421 138532(

98

76543219

−++++++++=

xxxxxxxxxSignY

);55553421 138532(

1098

765432110

−+++++++++=

xxxxxxxxxxSignY

);898955342113 8532(

1110987

65432111

−++++++++++=xxxxx

xxxxxxSignY

);14414489553421 138532(

12111098

765432112

−+++++++++++=

xxxxxxxxxxxxSignY

).233233144895534 21 138532(

1312111098

765432113

−++++++++++++=

xxxxxxxxxxxxxSignY

Table 3 contains bearing sets jT0 and jF0 for functions jY . In this table, combinations of variable values },...,,{ 21 nxxx correspond to decimal

equivalents of binary numbers 12

12 −

=∑ jj jx .



Table 3: Bearing sets for Horner's functions of the first type n T0 F0 8 85,86,88,96,128 63,79,83,84 9 171,172,176,192,256 127,159,167,169,170 10 341,342,344,352,384,512 255,319,335,339,340 11 683,684,688,704,768,1024 511,639,671,679,681,682 12 1365,1366,1368,1376,1408,1536,2048 1023,1279,1343,1359,1363,1364 13 2731,2732,2736,2752,2816,3072,4096 2047,2559,2687,2719,2727,2729,2730

5 Conclusion The proposed LTE has many attractive features. It is simple for hardware implementation in CMOS technology. Its β-comparator has very high sensitivity to current changes; this makes it possible to obtain the smallest voltage leap at the comparator output equal to 1V when the threshold of the realized function is 89 and to 325mV, if the threshold is 233. The implementability does not depend on the sum of input weights and is determined only by the function threshold. Such an LTE can perform very complicated functions, for example, logical threshold functions of 12 variables. The experiments confirm this result for functions of 10 variables. Moreover, during LTE learning all dispersions of technological and functional parameters of the LTE circuit are compensated. For enhancement of functional abilities a new circuit of the LTE synapse has been proposed, which gives to LTE the opportunity to have both excitatory and inhibitory inputs. The LTE with such synapses can be taught arbitrary threshold function of some number of variables in the case when it is not known beforehand which inputs are inhibitory and which are excitatory. An on-chip learning procedure, which allows the LTE to learn arbitrary threshold functions of 10 or less variables, has also been proposed. The solution is based on the well-known fact that any Boolean function of n variables can be represented as an isotonous function of 2n variables ( jx and jx ). The circuit in Fig.23 realizes this idea in the pure form. The function looks like

∑ ∑

∑ ∑

= =

= =

+

=−+=

n

j

n

jjjjj

n

j

n

jjjjj

xbxaRt

TxvxwSigny

1 1

1 1

)(

)(

where, if 0≠ja , then 0=jb and vice versa. The ability to determine the type of logical

variable inputs during the learning increases the number of realizable functions by 2n times

compared with the isotonous LTE implementation (n is the number of variables). We believe that the proposed LTE and its learning procedure can be very useful in many important applications including development of actual artificial neurons. The functional power of neurochips depends not only on the number of neurons that can be placed on one VLSI, but also on functional possibilities of a single neuron. It is evident that extending functional possibilities of a neuron is the prior aim when creating new neurochips, particularly in the case of a neuron implementation as a digital/analog circuit.

The main drawback of the proposed LTE is the high stability requirements for the supply voltage. This drawback appears to be peculiar to all circuits with high resolution, for example, digital-analog and analog-digital converters. It is natural to assume that rather slow changes of the supply voltage (with periods of not less than tens milliseconds) will be compensated during learning and refreshing. Its rapid changes can cause the loss of learned information. It would be reasonable to study the possibility of compensating the operational LTE parameters using circuit facilities.

Somebody can note that PSPICE simulations was oriented to very old CMOS technology of 0.8µm. Frankly speaking it does not matter what technology to use. We used old technology because of the problems stated in the paper have been studding during 15 years. There is no doubt that all simulations can be repeated for new technologies oriented on digital/analog implementations. Unfortunately submicron technologies are not appropriate because of big values of leakage currents. References: [1] C. Mead, Analog VLSI and Neural Systems, Addison-Wesley, 1989. [2] S.M. Fakhraie, K.C. Smith, VLSI-Compatible

Implementations for Artificial Neural



Networks, Kluwer, Boston-Dordrecht-London, 1997.

[3] T. Shibata, T. Ohmi, “Neuron MOS binary-logic integrated circuits: Part 1, Design fundamentals and soft-hardware logic circuit implementation”, IEEE Trans. on Electron Devices, Vol.40, No.5, 1993, pp. 974 – 979.

[4] T. Ohmi, T. Shibata, K. Kotani, “Four-Terminal Device Concept for Intelligence Soft Computing on Silicon Integrated Circuits”, Proc. of IIZUKA'96, 1996, pp. 49 – 59.

[5] T. Ohmi T “VLSI reliability through ultra clean processing”. Proc. of the IEEE, Vol.81, No.5, 1993, pp. 716-729

[6] V. Varshavsky, “Beta-Driven Threshold Elements”, Proceedings of the 8-th Great Lakes Symposium on VLSI, IEEE Computer Society, Feb. 19-21, 1998, pp. 52 – 58.

[7] V. Varshavsky, “Threshold Element and a Design Method for Elements”, filed to Japan's Patent Office, Jan.30, 1998, JPA H10-54079 (under examination).

[8] V. Varshavsky, “Simple CMOS Learnable Threshold Element”, International ICSC/IFAC Symposium on Neural Computation, Vienna, Austria, Sept.23-25, 1998.

[9] V. Varshavsky, “CMOS Artificial Neuron on the Base of Beta-Driven Threshold Elements”, IEEE International Conference on Systems, Man and Cybernetics, San Diego, CA, October 11-14, 1998, pp. 1857 – 1861.

[10] V. Varshavsky, “Synapse, Threshold Circuit and Neuron Circuit”, filed to Japan's Patent Office on Aug. 7, 1998, the application number is JPA-H10-224994.

[11] V. Varshavsky, “Threshold Element”, filed to Japan's Patent Office on Aug. 12, 1998, the application number is JPA-H10-228398.

[12] S. McCulloch, W. Pitts, “A Logical Calculus of the Ideas Imminent in Nervous Activity”, Bulletin of Mathematical Biophysics, 5, 1943, pp.115 – 133.

[13] F.E. Allen, D.R. Holberg, CMOS Analog Circuit Design, Oxford University Press, 1987.

[14] A. Montalvo, R. Gyurcsik, J. Paulos, “Toward a General-Purpose Analog VLSI Neural Network with On-Chip Learning” , IEEE Transactions on Neural Networks, Vol.8, No.2, March 1997, pp. 413 – 423.

[15] V. Varshavsky, V. Marakhovsky, “Learning Experiments with CMOS Artificial Neuron”, Lecture Notes in Computer Science 1625, ed. by Bernard Reusch, Computational Intelligence Theory and Applications. Proceedings of the 6th Fuzzy Days International Conference on Computational Intelligence, Dortmund, Germany, May 25-27, Springier, 1999, pp. 706 – 707.

[16] V. Varshavsky, V. Marakhovsky, “Beta-CMOS implementation of artificial neuron”, SPIE's 13th Annual International Symposium on Aerospace/Defense Sensing, Simulation, and Controls. Applications and Science of Computational Intelligence II, Orlando, Florida, April 5-8, 1999, pp. 210 – 221.

[17] V. Varshavsky, V. Marakhovsky, “Beta-CMOS Artificial Neuron and Implementability Limits”, Lecture Notes in Computer Science 1607, Jose Mira, Juan V. Sanchez-Andves (Eds), Engineering Applications of Bio-Inspired Artificial Neural Networks. Proceedings of International Work-Conference on Artificial and Natural Neural Networks (IWANN'99), Spain, June 2-4, Springier, Vol. 11, 1999, pp. 117 – 128.

[18] V. Varshavsky, V. Marakhovsky, “The Simple Neuron CMOS Implementation Learnable to Logical Threshold Functions”, Proceedings of International Workshop on Soft Computing in Industry (IWSCI'99), June 16-18, 1999, Muroran, Hokkaido, Japan, IEEE, 1999, pp. 463 – 468.

[19] V. Varshavsky, V. Marakhovsky, “Implementability Restrictions of the Beta-CMOS Artificial Neuron”, The Sixth International Conference on Electronics, Circuits and Systems (ICECS'99), Pafos, Cyprus, September 5-8, IEEE, 1999, pp. 401 – 405.



cmos implementation of an artificial neuron training on

Documents