2,737. v. - patentimages.storage.googleapis.com · "analog mos integrated circuits for signal...

United States Patent (19) Engeler

(54) NEURAL NET USING CAPACITIVE STRUCTURES CONNECTING OUTPUT LINES AND DIFFERENTALLY DRIVEN INPUT LINE PAIRS

William E. Engeler, Scotia, N.Y. General Electric Company, Schenectady, N.Y.

Appl. No.: 366,839 Filed: Jun. 15, 1989 Int. C. ....................... G06G 7/12; G06G 7/184;

G06J 1/00 U.S. C. ...................................... 395/24; 307/201;

364/602; 364/807 Field of Search ............... 364/807, 819, 825, 826,

364/827, 602, 604, 513; 307/201 References Cited

U.S. PATENT DOCUMENTS

3,890,635 6/1975 Engeler ................................. 357/14 3,950,733 4/1976 Cooper et al..... ... 364/200 3,969,636 7/1976 Baertsch et al. .. ... 307/304 X 4,156,284 5/1979 Engeler ............................... 364/862 4,288,858 9/1981 Merola et al. . ... 364/826 4,903,226 2/1990 Tsividis ............................... 364/807

OTHER PUBLICATIONS

Soclof, S., Analog Integrated Circuits, pub. by Pren tice-Hall, Inc., Englewood Cliffs, N.J. 07632, 1985, pp. 306, 333. Hosticka, B. et al., "MOS Sampled Data Recursive Filters Using Switched Capacitor Integrators", IEEE Trans on Solid-State Circuits, vol. SC-12, No. 6, Dec. 1977, pp. 600-608. Rumelhart, D. et al., Parallel Distributed Processing, vol. 1, copyright 1986 by The Massachusetts Inst. of Tech nology, pp. 45-54. Kub, F. et al., "Programmable Analog Synapses for Microelectronic Neural Networks Using a Hybrid Digital-Analog Approach', IEEE Int'l Conf. on Neu ral Networks, 1988. Bibyk, S., et al., "Issues in Analog VLSI and MOS Techniques for Neural Computing", in Analog VLSI Implementations of Neural Systems, ed. by C. Mead et al., copyright 1989 by Kluwer Academic Publishers,

75 Inventor: 73) Assignee:

(21) 22 (51)

(52)

(58)

56

US00546542A

11 Patent Number: 5,146,542 45 Date of Patent: Sep. 8, 1992

Proceedings of a workshop held May 8, 1989 in connec tion with the Intl. Symp. on Circuits and Systems, pp. 103-133. An Introduction to Computing With Neural Nets', R. P. Lippmann, IEEE ASSP Magazine, April 1987, pp. 4-22. "Switched-Capacitor Neural Networks", Y. P. Tsividis & D. Anastassiou, Electronics Letters, Aug. 27, 1987, vol. 23, No. 18, pp. 958-959. "Analog MOS Integrated Circuits for Signal Process ing', R. Gregorian & G. C. Temes, pp. 255-237, John Wiley & Sons, NYC, Chichester, Brisbane, Toronto, Singapore 1986. Electronics letters, vol. 23, No. 24, Nov. 19, 1987, (Ste venage, Herts, GB) Y. Tsividis et al.: "Analogue Cir cuits for Variable-Synapse Electronic Neural Net works," pp. 1313-1314. Electronics, vol. 45, No. 4, Feb. 14, 1972 R. G. Kos tanty: "Doubling op amp Summing Power", pp. 73-75, see FIG. 4; p. 75, paragraph: AC Considerations.

(List continued on next page.)

Primary Examiner-Stephen M. Baker Attorney, Agent, or Firm-Marvin Snyder; James C. Davis, Jr. 57 ABSTRACT Neural nets using capacitive structures are adapted for construction in complementary metal-oxide-semicon ductor integrated-circuit technology. Fully differential input amplifiers in each neural net layer apply their push-pull responses to synapse input signals via a pair of input lines. Each push-pull response is applied via a pair of capacitors of complementary capacitance values and an output line to the input of each of a plurality of non linear output amplifiers in that neural net layer, which generate respective axon responses for that neural net layer. In certain of these neural nets, arrangements are made such that the capacitive structures are bilaterally responsive so that back-propagation calculations can be performed during training periods, to alter the relative values of capacitors in each pair thereof.

57 Claims, 12 Drawing Sheets w

2,737. V. l, Ace) re: 1427 law gav

Aceos C C Nav Cem a Na

sie W Cy 9,(e) 32; a eia ca. > w)

22 NCa 942 Naga y m2 Cz,a ea o W > 2

3tag/ E. ae, W

else

C. C. ast Joe, Joyce, I. I.e.

5,146,542 Page 2

OTHER PUBLICATIONS

Proceedings Neural Networks from Models to Applica tions, ESPCI, Paris, 1988, H. P. Graf et al.: "VLSI Neural Network for Fast Pattern Matrching', pp. 725-732.

International Journal of Electronics, vol. 64, No. 3, Mar. 1988, (London, GB) A. Chichocki et al.: "Switch ed-capacitor Function Generators", pp. 359-375. Wescon 88, Conference Record, vol. 32, Nov. 1988, (North Hollywood, Calif., US), P. Hasler: "Implement ing practical Neural Networks in Silicon", pp. SS/1/1-6.

5,146,542 Sheet 1 of 12 Sep. 8, 1992 U.S. Patent

U.S. Patent Sep. 8, 1992 Sheet 2 of 12 5,146,542

8 VOO = -26/ g .2 AA/OA aa/

1- Ho Gy 96.

OM7 227 PGP9

AULLY OFAaceNTAL AMPL/Fea lss-ov B4s weTwoek


ss=OW N-N-1 N- -

WOW-L/WEAA MOZZAGA AOZZOWea 6/4S WeWORK

U.S. Patent Sep. 8, 1992

A/6/7AC CAAA6/7OA do, (Jew)

1. WOR0 S/OeAGE eleMewr Wise OFWrestral AM&MOQYAgeay

Sheet 4 of 12 5,146,542

CAAGE-SEMS/WG aMay/A/6e AQS

Ocfeaty

D/G/744. CAAA6/72a (M)(a)

Sheet 5 of 12 5,146,542 Sep. 8, 1992 U.S. Patent


fg. 5. A-A-70-7eMaaza, CO,

| A- A 70-746A/WAZ AA2 Coz Oz, O4 H -- is 70-7ew44 day '-Co

94 w-y R---70-7eMW44 day Cow-WPo OA He - Y - 70-76&M/WAL Day 'Coy

72 A. 7&644/WAZAZ. 4./


sszaawy/


xy7 7 az s7%, zdow6x


/* */99


E. CAOCA 00 GCL& C/Av GR COW7

acs,

(/p/OWCOWOZ

OAO Gop Couvr 9 ovr 1 Foegbow Cow

C/W

DC. Our

1.

NEURALNET USING CAPACTIVESTRUCTURES CONNECTING OUTPUT LINES AND

DFFERENTIALLY DRIVEN INPUT LINE PARS

The invention relates to computer structures that emulate portions of a brain in operation, and more par ticularly, to such computer structures as can be realized using complementary metal-oxide-semiconductor (CMOS) technology. RELATIONSHIP TO OTHER DISCLOSURES A patent application Ser. No. 366,838 concurrently

filed by the inventor, entitled NEURAL NET USING CAPACITIVE STRUCTURES CONNECTING INPUT LINES AND DIFFERENTIALLY SENSED OUTPUT LINE PAIRS and assigned to General Electric Company, discloses alternative neural net structures to those described herein.

BACKGROUND OF THE INVENTION

Computers of the von Neumann type architecture have limited computational speed owing to the commu nication limitations of the single processor. These limi tations may be overcome if a plurality of processors are utilized in the calculation and are operated at least partly in parallel. This alternative architecture, how ever, generally leads to difficulties associated with pro gramming complexity. Therefore, it is often not a good solution. Recently, an entirely different alternative that does not require programming has shown promise. The networking ability of the neurons in the brain has served as a model for the formation of a highly interconnected set of processors, called a "neural network' or "neural net" that can provide computational and reasoning functions without the need of formal programming. The neural nets can learn the correct procedure by experi ence rather than being preprogrammed for performing the correct procedure. The reader is referred to R. P. Lippmann's article "An Introduction to Computing With Neural Nets' appearing on pages 4-21 of the April 1987 IEEE ASSP MAGAZINE (0740-7467/87/0400-0004/$10.00" 1987 IEEE), incor porated herein by reference, for background about the state of the art in regard to neural nets.

Neural nets are composed of a plurality of neuron models, processors each exhibiting "axon' output signal response to a plurality of "synapse" input signals. In a type of neural net called a "perceptron', each of these processors calculates the weighted sum of its "synapse" input signals, which are respectively weighted by re spective weighting values that may be positive- or nega tive-valued, and responds non-linearly to the weighted sum to generate the "axon' output response. This rela tionship may be described in mathematical symbols as follows.

(1) M

y-f(3, WA) j = 1, 2, . . . N

Here, i indexes the input signals of the perceptron, of which there are an integral number M, andjindexes its N output signals, of which there are an integral number N. Wii is the weighting of the ilh input signal as makes up the jth output signal at such low input signal levels that the function

5

10

15

20

25

30

35

40

45

50

55

65

5,146,542

M f(;, PA) is approximately linear. At higher absolute values of its argument, the function

(g ) f X WXi is no longer exhibits linearity but rather exhibits a reduced response to

M

2, W.A. A more complex artificial neural network arranges a

plurality of perceptrons in hierarchic layers, the output signals of each earlier layer providing input signals for the next succeeding layer. Those layers preceding the output layer providing the ultimate output signal(s) are called "hidden" layers. The processing just described normally involves sam

pled-data analog signals, and prior-art neural nets have employed operational amplifiers with resistive intercon necting elements for the weighting and summing proce dures. The resistive elements implement weighted sum mation being done in accordance with Ohm's Law. The speed of such a processor is limited by capacitances in various portions of the processor, and computations have been slow if the power consumption of a reason ably large neural net is to be held within reasonable bounds. That is, speed is increased by reducing resis tance values to reduce RC time constants in the proces sors, but the reduced resistance values increase the V2/R power consumption (R, C and V being resistance, capacitance and voltage, respectively.) Using capaci tors to perform weighted summation in accordance with Coulomb's Law provide neural nets of given size operating at given speed that consume less power than those the processors which use resistors to implement weighted summation in accordance with Ohm's Law. Y. P. Tsividis and D. Anastassion in a letter "Switched Capacitor Neural Networks' appearing in ELEC TRONICS LETTERS, Aug. 27, 1987, Vol. 23, No. 18, pages 958,959 (IEE) describe one method of implement ing weighted summation in accordance with Coulomb's Law. Their method, a switched capacitor method, is useful in analog sampled-data neural net systems. How ever, a method of implementing weighted summation in accordance with Coulomb's Law that does not rely on capacitances being switched is desirable, it is here pointed out. This avoids the complexity of the capacitor switching elements and associated control lines. Fur thermore, operation of the neural net with continuous analog signals over sustained periods of time, as well as with sampled data analog signals, is thus made possible. A problem that is encountered when one attempts to

use capacitors to perform weighted summation in a neural net layer is associated with the stray capacitance between input and output lines, which tends to be of appreciable size in neural net layers constructed using a metal-oxide-semiconductor (MOS) integrated circuit technology. The input and output lines are usually laid out as overlapping column and row busses using plural

3 layer metallization. The column busses are situated in one layer of metallization and the row busses are situ ated in another layer of metallization separated from the other layer by an intervening insulating oxide layer. This oxide layer is thin so there is appreciable capaci tance at each crossing of one bus over another. The fact of the row and column busses being in different planes tends to increase stray capacitances between them. The stray capacitance problem is also noted where both row and column busses are situated in the same metallization layer with one set of busses being periodically inter rupted in their self-connections to allow passage of the other set of busses and being provided with cross-over connections to complete their self-connections. The problem of stray capacitance is compounded by the fact that the capacitive elements used to provide weights in a capacitive voltage summation network have stray capacitances to the substrate of the monolithic inte grated circuit in which they are incorporated; a perfect two-terminal capacitance is not actually available in the monolithic integrated circuit. Where capacitive ele ments having programmable capacitances are used, capacitance is usually not programmable to zero value, either. The foregoing problems with stray capacitance are

solved in the invention by using balanced input line pairs, so that the effects of stray capacitance from each output line tend to cancel out each other. Driving paired input lines with balanced input signals also al lows both excitory and inhibitory weights-that is, both positive- and negative-polarity Wi-in effect to be achieved without having to resort to capacitor switch ing in order to achieve negative capacitance. The Wixi terms are obtained by summing weighted xi and -xi balanced input signals.

Neural nets employing capacitors in accordance with the invention lend themselves to being used in perform ing parts of the computations needed to implement a back-propagation training algorithm. The back-propa gation training algorithm is an iterative gradient algo rithm designed to minimize the mean square error be tween the actual output of a multi-layer feed-forward neural net and the desired output. It requires continu ous, differentiable non-linearities. A recursive algorithm starting at the output nodes and working back to the first hidden layer is used iteratively to adjust weights in accordance with the following formula.

W(t+1)= W(t)-8px; (2)

In this equation Wi?(t) is the weight from hidden nodei (or, in the case of the first hidden layer, from an input node) to node j at time t; xi is the output signal of node i (which, in the case of the first hidden layer, an input signal); n is a gain term introduced to maintain stability in the feedback procedure used to minimize the mean square errors between the actual output(s) of the neural net and its desired output(s); and 8 is a derivative of error term for the nodej. The general definition of 8 is the change in error energy from output node j of a neural net layer with a change in the weighted summa tion of the input signals used to supply that output node J.

Lippman presumes that a particular sigmoid logistic non-linearity is used. Presuming the non-linearity of processor response is to be defined not as restrictively as Lippman does, then 8 can be more particularly defined as in equation (3), following, if nodej is an output node,

10

15

20

25

30

35

40

45

50

55

60

65

5,146,542 4.

or as in equation (4), following, if node j is an internal hidden node.

p y (4) 6 = Ji 6kWik

In equation (3) dandyjare the desired and actual values of output response from the output layer and yj is dif ferential response of yi to the non-linearity in the output layer-i.e., the slope of the transfer function of that non-linearity. In equation (4) k is over all nodes in the neural net layer succeeding the hidden node j under consideration and Wikis the weight between nodejand each such node k. The term yi is defined in the same way as in equation (3). The general definition of the yf term appearing in

equations (3) and (4), rather than that general term being replaced by the specific value of y, associated with a specific sigmoid logistic non-linearity, is the primary difference between the training algorithm as described here and as described by Lippmann. Also, Lippman defines 8 in opposite polarity from equations (l), (3) and (4) above.

During training of the neural net prescribed patterns of input signals are sequentially repetitively applied, for which patterns of input signals there are corresponding prescribed patterns of output signals known. The pat tern of output signals generated by the neural net, re sponsive to each prescribed pattern of input signals, is compared to the prescribed pattern of output signals to develop error signals, which are used to adjust the weights per equation (2) as the pattern of input signals is repeated several times, or until the error signals are detected as being negligibly valued. Then training is done with the next set of patterns in the sequence. Dur ing extensive training the sequence of patterns may be recycled.

SUMMARY OF THE INVENTION The invention generally concerns neural nets the

processors of which use capacitors to perform weighted summation in accordance with Coulomb's Law. Each processor includes an output line on which the weighted summation of input signal voltages appears and a non-linear amplifier for providing neuron-like response to that weighted summation. A plurality, 2M in number, of input lines, crossing by that output line and exhibiting a stray capacitance thereto, are each connected to that output line by a respective capacitive element. Each input line and the capacitive element connecting it to the output line are identified by a re spective one of consecutive ordinal numbers first through 2M', M being a positive integer. Means are provided for applying M input signal voltages in bal anced form each to a respective pair of input lines iden tified by ordinal numbers M apart. The differences in the capacitances of the capacitive elements connecting to the output line from each pair of input lines deter mines the weighting of the input signal applied in bal anced form on that pair of input lines, as appears in the weighted summation of input signal on the outline line. Accordingly, the effects of stray capacitance on the weighted summation of input signals is reduced.

5,146,542 5

BRIEF DESCRIPTION OF THE DRAWING

FIG. 1 is a schematic diagram of a neural net layer which embodies the invention, using capacitors to per form weighted summations of synapse signals to be subsequently non-linearly amplified to generate axon response signals. FIG. 2 is a schematic diagram of a prior art fully

differential amplifier and a bias network therefore, as implemented with complementary metal-oxide-semi conductor CMOS field effect transistors, which is use ful in the construction of neural nets in accordance with the invention.

FIG. 3 is a schematic diagram of a non-linear voltage amplifier that is useful in the construction of neural nets in accordance with the invention. FIGS. 4A and 4B together form a FIG. 4 that is a

schematic diagram of a modification of the FIG. 1 neu ral net that can be made manifold times to provide, in accordance with a further aspect of the invention, for the programmable weighting of the capacitances used in performing weighted summation of synapse signals.

Each of FIGS. 5 and 6 is a schematic diagram illus trating one way of pulsing the non-linear output driver amplifiers in a FIG. 1 neural net layer modified mani foldly per FIG. 4.

FIG. 7 is a schematic diagram of training apparatus used with the FIG. 1 neural net layer manifoldly modi fied per FIG. 4.

FIG. 8 is a schematic diagram of a system having a plurality of neural net layers each constructed in accor dance with FIG. 1 modified manifold times per FIG. 4. FIGS. 9A and 9B together form a FIG. 9 that is a

schematic diagram of an alternative modification of the FIG. 1 neural net that can be made manifold times to provide during training for the programmable weighting of the capacitances used in performing weighted summation of synapse signals, in accordance with another aspect of the invention.

FIG. 10 is a schematic diagram of the arrangement of stages in each counter of the FIG. 1 neural net modified per FIG. 9.

FIG. 11 is a schematic diagram of the logic elements included in each counter stage.

FIG. 12 is a schematic diagram of further modifica tions that can be made to the neural net layers in further embodiments of the invention, which further modifica tion halves the number of connections between neural net layers and reduces the number of fully differential amplifiers required in each neural net layer.

DETALED DESCRIPTION

FIG. 1 shows a neural net comprising a plurality, Nin number, of non-linear amplifiers OD, OD2, . . . OD(N-1), ODN supplied respective input voltages from apparatus for matrix multiplying as described by the inventor in U.S. Pat. No. 4,156,284 issued May 22, 1979, entitled SIGNAL PROCESSING APPARATUS' and assigned to General Electric Company. The appa ratus for matrix multiplying weights each of a plurality, M in number, of input voltage signals x1, x2, ... x(M-1),

O

15

20

25

30

35

40

45

50

55

XM supplied as "synapse' signals to provide the respec tive input voltages for the non-linear voltage amplifiers OD1, OD2, ... ODN-1, ODN, which generate respec tive "axon" responses y1, y2, ... Yon-1), yN. M is a positive plural integer indicating the number of

input synapse signals to the FIG. 1 net, and N is a posi tive plural integer indicating the number of output axon

6 signals the FIG. 1 net can generate. To reduce the writ ten material required to describe operation of the FIG. 1 neural net, operations using replicated elements will be described in general terms; using a subscripti ranging over all values one through M for describing operations and apparatuses as they relate to the (column) input signals x1, x2, ... x(M-1), XM, and using a subscript j ranging over all values one through N for describing operations and apparatus as they relate to the (row) output signals y1, y2, ... y(N-1), YN. That is, i.and j are the column and row numbers used to describe particu lar portions of the neural net.

Input voltage signal x is applied to the input port of an input driver amplifier ID that in turn applies its non-inverted voltage response from its (--) output port to an input line ILi and applies its inverted voltage re sponse from its (-) output port to an input line IL(i-M). A respective output line OL connects to the input port of output driver amplifier OD, which generates at its output port a nonlinear voltage response to the cumula tive charge on that respective output line OLi.

FIG. 2 shows a fully differential amplifier con structed of MOS field-effect transistors Q1-Q13, as may serve for any one of the input driver amplifiers ID for i-1,2,... M. Also shown is a bias network constructed of MOS field effect transistors Q14-Q19 for generating direct bias potentials for application to that fully differ ential amplifier and to others of its kind. This circuitry is described in more detail on pages 255-257 of the book Analog MOS Integrated Circuits for Signal Processing by R. Gregorian and G. C. Temes, copyright 1986 by John Wiley & Sons, Inc., of New York, Chichester, Brisbane, Toronto and Singapore. The fully differential amplifier includes a long-tailed

pair connection of n-channel MOSFETs Q1 and Q2 providing common-mode rejection for the input volt ages IN and IN applied to the (--) and (-) input termi nals at their respective gate electrodes. N-channel MOSFET Q13 is connected as a constant-current sink for tail current from the interconnection between the source electrodes of Q1 and QL. Q1 and Q2 are in folded cascode connections with p-channel MOSFETs Q7 and Q8 respectively. There is also common mode rejection for output voltages OUT and OUT appearing at the (--) and (-) output terminals connecting from the drain electrodes of Q7 and Q8 respectively, which is why the differential amplifier comprising QF-Q13 is described as being "fully differential. This common mode rejection is provided by common-mode degenerative feedback connections from the (-) and (--) output terminals to the gate electrodes of p-channel MOSFETs Q3 and Q4, the paralleled source-to-drain paths of which supply current to the joined source electrodes of p-channel MOSFETs Qs and Q6 operated as a current splitter. Qs drain current biases the folded cascode connection of Q1 and Q7, and Q6 drain current biases the folded cas code connection of Q2 and Q8. N-channel MOSFETs Q9 and Q11 are in a cascode connection biased to pro vide a high-impedance constant-current sink as drain load to Q7, and n-channel MOSFETs Q10 and Q12 are in a cascode connection biased to provide a high-imped ance constant-current sink as drain load to Q8. The (--) and (-) output terminals can be biased to

the same (+2.5 v) potential as applied to the gate elec trode of MOSFET Q14 by causing MOSFETs Q1-Q18 to have the following width-to-length ratios presuming Q1, Q2, Q7 and Q8 to have equal amplitude quiescent channel currents.

5,146,542 7

(W/L)11: (W/L)12: (W/L)13: (W/L)18: 2:2::1 (5) (W/L)3: (W/L)4: (W/L)14: 1:1:1 (6) (W/L).5: (W/L)6: (W/L).5: 1:1: (7) (W/L)7: (W/L)8: (W/L)6: 2:2:1 (8) (W/L).9: (W/L)10: (W/L)7: 2:2:1 (9)

The width-to-length ratio of MOSFET Q19 is chosen to provide responsive to drain current demand of QW14 a voltage drop across Q19 channel that affords a sufficient operating range for signals at terminals OUT and OUT.

This proportioning of Q1-Q18 and a respective degen erative feedback connection from its (--) output termi nal to its (-) input terminal conditions each of the input driver amplifiers ID in the FIG. 1 neural net to provide xi voltage-follower response at its (--) output terminal to xi signal applied to its (--) input terminal and to pro vide inverted, -Xi response at its (-) output terminal. The non-linear output driver amplifier OD, is shown

in FIG. 1 as being just a non-linear voltage amplifier with the quiescent direct potential applied to its input signal terminal via output line OL being adjusted by clamping to a desired bias voltage at selected times using a respective direct-current restorer circuit DCR. The restorer circuit DCR is shown separated from non-linear output driver ODjin FIG. 1 to improve the layout of the drawing, but customarily is more closely associated with the output driver OD, in accordance with customary practice in dc restoration. A respective capacitor Ciconnects each of the input lines IL to each of the output lines OL, and a respective capacitor Co-M) connects to each of the output lines OL, the one of the input lines ILGM) paired with that ILi. Since the paired ILi and IL(i-M) input lines are driven with xi and -Xi signal voltages respectively, the electrically equiva lent circuit is xi signal voltage being applied to output line OL by a capacitor having a capacitance that equals the capacitance of Ciminus the capacitance of CGM). This balanced input signal drive to paired input lines technique avoids the need for switched-capacitance techniques in order to obtain inhibitory as well as exci tory weights, and thus facilitates operating the neural net with analog signals that are continuous over sus tained periods of time, if so desired. FIG. 1 shows each of the input lines ILior IL(M) as

being provided with a respective load capacitor CL or CL(i+M) to cause that capacitive loading upon each of the output terminals of the input driver amplifier ID to be substantially the same as that upon each output port of the other input driver amplifiers. This is desirable for avoiding unwanted differential delay in responses to the input signals xi. Substantially equal capacitive loading can be achieved by making the capacitance of each of the input line loading capacitors CLI-CL2M very large compared to the total capacitance of the capacitors Ci or C(i+M) connecting thereto. Preferably, however, this result is achieved by making the capacitance of each of the input line loading capacitors complement the combined value of the other capacitances connect ing thereto. This procedure reduces the amount of line loading capacitance required. Where the voltage ap pearing on the output lines is sensed directly by the non-linear output driver amplifiers OD1, . . . ODN, as shown in FIG. 1, this preferable procedure makes the voltage division ratio for each input voltage xi, ... x M independent of the voltage division ratios for the other input voltages. Where the charge appearing on the output lines is sensed by charge-sensing amplifiers pre

10

15

20

25

30

35

45

50

55

65

8 ceding the non-linear output driver amplifiers, as will be described later on in this specification in connection with FIG. 4, this latter consideration is not as impor tant.

FIG. 1 also shows each of the output lines OLibeing loaded with a respective load capacitor CL(2M) to cause the total capacitance on that line to remain sub stantially the same as on each of the other output lines. Again, this can be done either by choosing CL(2M) to be much larger than other capacitances to output line OL, or by choosing CL(2M) to complement the com bined value of the other capacitances connecting thereto. The input voltage to output driver amplifier OD will (to good approximation) have the following value, vi, in accordance with Coulomb's Law.

(10) y = (C), (C - Ca+ M),j}x

Here Cis the total capacitance on output line OL. The generation of voltage vican be viewed as the superposi tion of a plurality of capacitive divisions between, on the one hand, the effective capacitance (Co-C-M)) each input voltage has to output line OLi and, on the other hand, the total capacitance Cof the output line to its surroundings.

FIG. 3 shows non-linear voltage amplifier circuitry that can be used after linear voltage amplifier circuitry to implement each non-linear output driver amplifier ODjin the FIG. 1 neural net layer. The FIG.3 non-lin ear voltage amplifier is a cascade connection of two source-follower transistors, one (Q20A) being an n-chan nel MOSFET and the other (Q20B) being a p-channel MOSFET. Q20A is provided a constant-current genera tor source load by an n-channel MOSFET Q21, which is the slave or output transistor of a current mirror ampli fier including as its master or input transistor an n-chan nel MOSFET Q22 self-biased by drain-to-gate feedback. Q20B is provided a constant-current generator source load by a p-channel MOSFET Q23, which is the slave or output transistor of a current mirror amplifier including as its master or input transistor a p-channel MOSFET Q24 self-biased by drain-to-gate feedback. Q22 and Q24 are connected as diodes by their respective drain-to gate feedback connections, and these diodes are con nected in series with another diode-connected n-chan nel MOSFET Q25 and with another diode-connected p-channel MOSFET Q26 between VSS and VDD poten tials to implement a bias network. In this bias network a quiescent input current flows from the input port of the current mirror amplifier comprising Q23, Q24 into the input port of the current mirror amplifier comprising Q21, Q22. Q2 and Q23 drain current flows are similar valued by current mirror amplifier action.

All the n-channel MOSFETs Q204, Q21, Q22 and Q25 have similar channel widths and lengths and exhibit similar operating characteristics. All the p-channel MOSFETs Q20B, Q23, Q24 and Q26 have similar channel widths and lengths and exhibit similar operating charac teristics, which are complementary to those of the n channel MOSFETs. The bias network MOSFETs Q2, Q24, Q25 and Q26 may be shared by a plurality of the FIG. 3 non-linear voltage amplifier circuits, to conserve hardware and operating power. m

Non-linearity of response in the FIG. 3 voltage am plifier comes about because (1) source-follower action of Q20A for positive-going excursions of its gate elec

5,146,542 trode potential becomes limited as its source potential approaches its drain potential VHI and (2) source-fol lower action of Q20B for negative-going excursions of its gate electrode potential becomes limited as its source potential approaches its drain potential VLo. At the source electrode of source-follower Q20B there is a sig moidal response to a linear ramp potential applied to the gate electrode of source-follower Q204. The voltages VLoand VH1 can be programmed to control the limiting properties of the FIG. 3 non-linear amplifier, and the voltages VLo and VHF may be selected to provide for symmetry of response or for asymmetry of response. FIG. 3 shows representative values for VHF and VLo that provide a substantially symmetrical response about --2.5 volts. Output driver amplifier OD can use non-linear volt

age amplifier circuitry different from that shown in FIG. 3. For example, source followers Q204 and Q20B can be reversed in order of their cascade connection. Either this alternative circuitry or the FIG. 3 circuitry can be preceded by a charge-sensing amplifier, rather than a linear voltage amplifier, to realize the type of output driver amplifier used in FIG. 4 and FIG.9 neural nets. In the FIG. 1 neural net the output driver amplifi ers can be realized without using the FIG. 3 circuitry or the previously described alternative circuitry. For ex ample, each output driver amplifier can comprise a long-tailed pair connection of transistors having a cur rent mirror amplifier load for converting their output signal voltage to single-ended form. The long-tailed pair connection of transistors is a differential amplifier connection where their source electrodes have a differ ential-mode connection to each other and to a constant current generator.

Consider now how neuron model behavior is exhib ited by input driver amplifier ID, capacitors Ci and Co-M), and non-linear output driver amplifier OD, for particular respective values of i and j. The voltage re sponses input driver amplifier ID applies to input lines ILi and ILGEM) are the same in amplitude but are of opposing polarity as referred to a common-mode volt age that is designed to be nominally the same as a bias voltage VBIAS midway between the 0-volt VSS and --5- volt VDD supply voltages. If the capacitance of capaci tor Ci is larger than the capacitance of capacitor Co-M) for these particular values of i and j, then the output voltage y for that j will exhibit "excitory" re sponse to the input voltage xi. If the capacitances of Ci and CG-M) are equal for these i and j values, then the output voltage y for thatjshould exhibit no response to the input voltage yj. If the capacitance of capacitor Ci is smaller than the capacitance of capacitor Coi--M) for those i and j values, then the output voltage y for that j will exhibit "inhibitory' response to the input voltage Xi.

In some neural nets constructed in accordance with the invention the capacitors Ci and Co-M) for alli and j may be fixed-value capacitors, so there is never any alteration in the weighting of input voltages xi where i=1,... M. However, such neural nets lack the capac ity to adapt to changing criteria for neural responses-- which adaptation is necessary, for example, in a neural network that is to be connected for self-learning. It is desirable in certain applications, then, to provide for altering the capacitances of each pair of capacitors Cit and C(Miassociated with a respective pair of values ofiandj. This alteration is to be carried out in a comple mentary way, so the sum of the capacitances of Cili and

10

15

20

25

30

35

40

45

50

55

65

10 of CGM) remains equal to CK. For example, this can be implemented along the lines of the inventor's previous teachings in regard to "digital' capacitors, having ca pacitances controlled in proportion to binary-numbers used as control signals, as particularly disclosed in con nection with FIG. 11 of his U.S. Pat. No. 3,890,635 issued Jun. 17, 1975, entitled VARIABLE CAPACI TANCE SEMCONDUCTOR DEVICES' and as signed to General Electric Company. Each pair of ca pacitors Ciand CG-M) is then two similar ones of these capacitors and their capacitances are controlled by respective control signals, one of which is the one's complement of the other.

Alternatively, the pair of capacitors Ci and Coi--M), may be formed from selecting each of a set of compo nent capacitors with capacitances related in accordance with powers of two to be a component of one or the other of the pair of capacitors Ci and Co-M), the selecting being done by field effect transistors operated as transmission gates. Yet another way of realizing the pair of capacitors Ci and C(-M) is to control the in verted surface potentials of a pair of similar size metal oxide-semiconductor capacitors with respective analog signals developed by digital-to-analog conversion.

FIG. 4, comprising component FIGS. 4A and 4B, shows a representative modification that can be made to the FIG. 1 neural net near each set of intersections of an output line OLi with input lines ILi and ILG-M) driven by opposite senses of a synapse input signal xi. Such modifications together make the neural net capable of being trained. Each capacitor pair Ci and C(i+M) of the FIG. 1 neural net is to be provided by a pair of digital capacitors DCi and DC(+M). (For example, each of these capacitors DCi and DCG-M may be as shown in FIG. 11 of U.S. Pat. No. 3,890,635). The ca pacitances of DCi and DC(-Mare controlled in com plementary ways by a digital word and its one's comple ment, as drawn from a respective word-storage element WSEijin an array of such elements located interstitially among the rows of digital capacitors and connected to form a memory. This memory may, for example, be a random access memory (RAM) with each word-storage element WSEij being selectively addressable by row and column address lines controlled by address decod ers. Or, by way of further example, this memory can be a plurality of static shift registers, one for each column j. Each static shift register will then have a respective stage WSEij for storing the word that controls the ca pacitances of each pair of digital capacitors DCi and DC(i+M). The word stored in word storage element WSEijmay

also control the capacitances of a further pair of digital capacitors DCi 0+N) and DC(i+M),(i+N), respectively. The capacitors DCi 0+N) and DC(i+M),(i+N) connect between "ac ground" and input lines ILi and ILGM), respectively, and form parts of the loading capacitors CLi and CLG-FM), respectively. The capacitances of DC(i+M,0+N) and DCi are similar to each other and changes in their respective values track each other. The four digital capacitors DC, DC(i+M), j, DC,0+N) and DC(i+M)0+N) are connected in a bridge configuration having input terminals to which the input lines ILi and ILGM) respectively connect and having output termi nals connecting to output line OL and to ac ground respectively. The capacitances of DCO-N) and DCGM) are similar to each other and changes in their respective values track each other. This bridge configu ration facilitates making computations associated with

5,146,542 11

back-propagation programming by helping make the capacitance network bilateral insofar as voltage gain is concerned. Alternatively, where the computations for back-propagation programming are done by computers that do not involve the neural net in the computation procedures, the neural net need not include the digital capacitors DC-N and DC(i+M),0+N). These digital capacitors DCO-N) and DC(1+M),0+N) are not needed either where very large loading capacitors are placed on the output lines OL, but this alternative undesirably reduces sensitivity of the output driver amplifier OD When the FIG. 4 neural net is being operated nor

mally, following programming, the designal applied to a mode control line MCL, is a logic ZERO. This ZERO conditions a respective input line multiplexer ILM to connect the non-inverting output port at each input driver amplifier ID; to input line ILi. The designal on mode control line MCL being a ZERO also conditions a respective input line multiplexer ILM(i+M) to connect the inverting output port of each input driver amplifier IDi to input line IL-(+M). A ZERO on mode control line MCL also conditions

each output line multiplexer OLM of an n-numbered plurality thereof to select the output line OLi to the inverting input terminal of a respective associated dif ferential-input amplifier DA, included in a respective charge-sensing amplifier QS that performs a charge sensing operation for output line OL. In furtherance of this charge-sensing operation, a transmission gate TG responds to the absence of a reset pulse QR to connect an integrating capacitor CI between the output and inverting-input terminals of differential-input amplifier DA. Amplifier DAjmay be an operational amplifier of the conventional voltage amplifier type or may be an operational transconductance amplifier. With integrat ing capacitor Ciso connected, amplifier DAi functions as a charge amplifier. When designal on mode control line MCLis a ZERO, the input signal x induces a total change in charge on the capacitors DCi and DC(i+M) proportional to the difference in their respective capaci tances. The resulting displacement current flow from the inverting input terminal of differential-input ampli fier DArequires that there be a corresponding displace ment current flow from the integrating capacitor CI charging that capacitor to place thereon a voltage vi defined as follows.

- (11) vj = (CI i (Cij - Ci--M))xi

The voltage V is supplied to a non-linear voltage amplifier circuit NL, which can be the non-linear volt age amplifier circuit of FIG. 3 or an alternative circuit as previously described. The non-linear voltage ampli fier circuit responds to generate the axon output re sponse Yi. From time to time, the normal operation of the neural

net is interrupted, and to implement dic-restoration a reset pulse dR is supplied to each charge sensing ampli fier QSi Responsive to d5R the logic compliment of the reset pulse dbR, going low when dR goes high, transmis sion gate TG is no longer rendered conductive to con nect the integrating capacitor CI from the output termi nal of differential amplifier DA. Instead, a transmission gate TG0+N) responds to dbR going high to connect to VBiAs the plate of capacitor Ci normally connected from that output terminal, VBIAS being the 2.5 volt intermediate potential between the Vss=0 volt and

10

15

20

25

30

35

40

45

50

55

65

12 VDD=5 volt operating voltages of differential amplifier DA. Another transmission gate TG(j+2N) responds to dbr going high to apply direct-coupled feedback from the output terminal of differential amplifier DA to its inverting input terminal, to bring the voltage at the output terminal to that supplied to its inverting input terminal from output line OL. During the dc-restora tion all xi are "zero-valued". So, the charge on integrat ing capacitor CI is adjusted to compensate for any direct voltage error occurring in the circuitry up to the output terminal of differential amplifier DA DC-resto ration is done concurrently for all differential amplifiers DA (i.e., for values of jranging from one to N). During training, the dip signal applied to mode con

trol line MCL is a logic ONE, which causes the output line multiplexer OLM, to disconnect the output line OL from the inverting input terminal of differential amplifier DA, and to connect the output line OLi to receive a 8terror term. This 8error term is generated as the product output signal of a analog multiplier AM, responsive to a signal Ai and to a signal yi which is the change in output voltage yi of non-linear amplifier NLj for unit change in the voltage on output line OLj. The term A is for the output neural net layer the difference between yi actual value and its desired value di. The term Ajis for a hidden neural net layer the Ai output of the succeeding neural net layer during the back-propa gation procedure.

Differentiator DF generates the signal yi, which is a derivative indicative of the slope of yjchange in voltage on output line OL superposed on VBIAs. To determine the y' derivative, a pulse doublet comprising a small positive-going pulse immediately followed by a similar amplitude negative-going pulse is introduced at the inverting input terminal of differential amplifier DA (or equivalently, the opposite-polarity doublet pulse is in troduced at the non-inverting input terminal of differen tial amplifier DA) to first lower yi slightly below nor mal value and then raise it slightly above normal value. This transition of y from slightly below normal value to slightly above normal value is applied via a differentiat ing capacitor CD to differentiator DFj.

Differentiator DFincludes a charge sensing amplifier including a differential amplifier DA(i+N) and an inte grating capacitor CI0+N). During the time that yj is slightly below normal value, a reset pulse dbs is applied to transmission gates TG(j-4N) and TG(j-5N) to render them conductive. This is done to drain charge from integrating capacitor CIJN), except for that charge needed to compensate for DA(i+N) input offset voltage error. The reset pulse dbs ends, rendering transmission gates TGB6-4N) and TG(j-5N) no longer conductive, and the complementary signal dbs goes high to render a transmission gate TG-3N) conductive for connecting integrating capacitor CI(i+N) between the output and inverting-input terminals of differential amplifier DA(i+N). With the charge sensing amplifier comprising ele

ments DA(i+N) and CIG+N) reset, the small downward pulsing of y from normal value is discontinued and the small upward pulsing of yi from normal value occurs. The transition between the two abnormal conditions of yi is applied to the charge sensing amplifier by electro static induction via differentiating capacitor CD. Dif. ferential amplifier DA(i+N) output voltage changes by an amounty' from the VBIAs value it assumed during reset. The use of the transition between the two pulses

5,146,542 13

of the doublet, rather than the edge of a singlet pulse, to determine the derivative y' makes the derivative-taking process treat more similarly those excitory and inhibit ing responses of the same amplitude. The doublet pulse introduces no direct potential offset error into the neu ral net layer. Responsive to a pulse dT, the value yj +VBIAS from

differentiator DF is sampled and held by (row) sample and hold circuit RSH, for application to analog multi plier AM, as an input signal. This sample and hold pro cedure allows yi to return to its normal value, which is useful in the output layer to facilitate providing yi for . calculating (y-d). The sample and hold circuit RSH may simply comprise an L-section with a series-arm transmission-gate sample switch and a shunt-leg hold capaicitor, for example. Analog multiplier AM is of a type accepting differential input signals--e.g., as de scribed by K. Bullt and H. Wallinga in their paper A CMOS Four-quadrant Analog Multiplier" appearing on pages 430-435 of the IEEE JOURNAL OF SOLID STATE CIRCUITS, Vol. SO-21, No. 3, June 1986, incorporated herein by reference. The difference be tween Vi+VBLAS and VBIAs voltages is used as a differ ential input signal to analog multiplier AM, which exhibits common-mode rejection for the VBIAS term. During training, the designal applied to the mode

control line MCL is a ONE, as previously noted, and this causes the input line multiplexers ILM and ILM (i+ M) to disconnect the input lines ILi and IL(i+M) from the input driver amplifier ID, output terminals and con nect them instead to the non-inverting and inverting input terminals of a differential charge-sensing amplifier DQSi. The voltage 8 induces a differential change in charge between input lines IL and IL(i+M) proportional to 8, (C-C(i+ M)), which differential change in charge is sensed using the differential charge sensing amplifier DQSi.

Differential charge-sensing amplifier DQS includes a fully differential amplifier (as shown in FIG. 2, for ex ample) provided with integrating capacitors IC and IC(i+M) in respective degenerative feedback connec tions from each of its output terminals to each of its input terminals. Resetting of differential charge-sensing amplifier DQS is similar to the resetting of a single ended amplifier such as QS except for involving two integrating capacitors ICi and IC(in), rather than just the one integrating capacitor C. Resetting of differen tial charge-sensing amplifier DQS is done responsive to a pulse db U, which occurs during the time when mode control line MCL has a ONE thereon conditioning input line multiplexers ILMi and IMG-m) to connect input lines ILi and IL(i-n) differential charge-sensing amplifier DQS. Resetting is normally done shortly after a ZERO to ONE transition appears in the designal applied to mode control line MCL and may also be done at other times. This procedure corrects for capaci tive unbalances on the input lines ILi and IL(i+M) during back-propagation computations that follow the reset ting procedure. In these computations voltages --Ai-ii-VBIAS and -AVBIAS are developed at the (--) and (-) output terminals of the fully differential ampli fier included in differential charge-sensing amplifier DQSi. The voltage --Ai--VBLAS is used by the preced ing neural net layer during the back-propagation train ing procedure, if such a preceding neural net layer ex ists. The use of single-ended --Ali and + A drive is shown in FIG. 4, presuming the neural net layers to be integrated within separate monolithic integrated cir

10

5

20

25

30

35

45

50

55

65

14 cuits, and presuming the limitation on number of pin outs is restrictive. Where a plurality of neural net layers are integrated within the same monolithic integrated circuitry, or where maximum pin-out count is not a restrictive design factor, balanced A signals may be applied from one neural net layer to the preceding one. So, too, if the non-linear voltage amlifier NL is of a correct type (for example, a long-tailed pair connection of transistors) yi-I-VBIAs and -yi+VBLAs balanced out put signals may be supplied to the next neural net layer. Indeed, the y' signals applied to the analog multiplier AM may be generated in balanced form, replacing differentiator DF, and sample-and-hold circuit SH with balanced circuitry. FIG. 5 shows how each output line OL, for j=1,...

N may be pulsed during calculation of y' terms. Each output line OL is connected by a respective capacitor CO, to the output terminal of a pulse generator PG, which generates the doublet pulse. FIG. 5 shows the doublet pulse applied to the end of each output line OL remote from the -terminal of the associated differential amplifier DA in the charge-sensing amplifier QS sens ing the charge on that line. It is also possible to apply the doublet pulses more directly to those (-) terminals by connecting to these terminals respective ones of the plates of capacitors CO that are remote from the plates connecting to pulse generator PG.

FIG. 6 shows how the non-inverting (--) input termi nal of each differential amplifier DA may be pulsed during calculation of y' terms, rather than pulsing the OL output lines. VBIAs, rather than being applied di rectly to the non-inverting input terminals of each dif ferential amplifier DA, is applied from the output ter minal of a biasing operational amplifier BOA. Opera tional amplifier BOA has its non-inverting terminal connected to receive VBIAS, and has degenerative feed back applied via a resistor R1 from its output terminal to its inverting input terminal. This degenerative feed holds the quiescent level of operational amplifier BOA output terminal at VBIAs. A pulse generator PG applies a doublet pulse via the series connection of a resistor R2 having resistance and a dc-blocking capacitor CB to the inverting input terminal of operational amplifier BOA. Presuming the resistances of resistors R1 and R2 to be R1 and R2, respectively, a doublet pulse -R1/R2 times the amplitude of the doublet pulse supplied from the pulse generator PG is applied from the output terminal of operational amplifier DBA to the non-inverting input terminals of differential amplifiers DA. This doublet pulse, having no direct voltage component, does not alter the VBIAS direct component maintained at the output terminal of operational amplifier BOA by the degenerative feedback connection back to the inverting input connection of BOA. Arrangements for adding the doublet pulse to vibe

fore its application to the non-linear amplifier NL can be used, rather than using the FIG. 5 or FIG. 6 arrange ment. -

FIG. 7 shows apparatuses for completing the back propagation computations, as may be used with the FIG. 1 neural net manifoldly modified per FIG. 4. The weights at each word storage element WSE in the interstitial memory array IMA are to be adjusted as the icolumn addresses andjrow addresses are scanned row by row, one column at a time. An address scanning generator ASG generates this scan of i and j addresses shown applied to interstitial memory array IMA, as suming it to be a random access memory. The row

5,146,542 15

address j is applied to a row multiplexer RM that selects 8 to one input of a multiplier MULT, and the column address i is applied to a column multiplexer CM that selects xi to another input of the multiplier MULT.

Multiplier MULT is of a type providing a digital output responsive to the product of its analog input signals. Multiplier MULT may be a multiplying analog to-digital converter, or it may comprise an analog multi plier followed by an analog-to-digital converter, or it may comprise an analog-to-digital converter for each of its input signals and a digital multiplier for multiplying together the converted signals. Multiplier MULT gen erates the product xi 8, as reduced by a scaling factor m, which is the increment or decrement to the weight stored in the currently addressed word storage element WSE in the memory array IMA. The former value of weight stored in word storage element WSEii is read from memory array IMA to a temporary storage ele ment, or latch, TS. This former weight value is supplied as minuend to a digital subtractor SUB, which receives as subtrahend m x 8 from multiplier MULT. The result ing difference is the updated weight value, which is written into word storage element WSEi in memory array IMA to replace the former weight value.

FIG. 8 shows how trained neural net layers Lo, Ll and L2 are connected together in a system that can be trained. Lois the output neural net layer that generates Youtput signals, is similar to that described in connec tion with FIGS. 4 and 5, and is provided with a back propagation processor BPPo with elements similar to those shown in FIG. 6 for updating the weights stored in the interstitial memory array of L0. L1 is the first hidden neural net layer which generates yi output sig nals supplied to the output neural net layer as its xiinput signals. These yi output signals are generated by layer L1 as its non-linear response to the weighted sum of its xh input signals. This first hidden neural net layer L1 is provided with a back-propagation processor BPP simi lar to BPPo. L2 is the second hidden neural net layer, which generates yh output signals supplied to the first hidden neural net layer as its xh input signals. These yh output signals are generated by layer L2 as its non-linear response to a weighted summation of its xginput signals. This second hidden layer is provided with a back propagation processor similar to BPPo and to BPP1.

FIG. 8 presumes that the respective interstitial mem ory array IMA of each neural net layer Lo, Li, L2 has a combined read/write bus instead of separate read input and write output busses as shown in FIG. 7. Back propagation processor BPPo modifies the weights read from word storage elements in neural net layer Lointer stitial memory array by m x 6amounts and writes them back to the word storage elements in a sequence of read-modify-write cycles during the training procedure. Back-propagation processor BPP modifies the weights read from word storage elements in neural net layer L1 interstitial memory array by mixh 8 amounts and writes them back to the word storage elements in a sequence of read-modify-write cycles, during the training proce dure. Back-propagation processor BPP2 modifies the weights read and storage elements in neural net layer L2 interstitial memory array by m xg 5h amounts and writes them back to the word storage element in a se quence of read-modify-write cycles during the training procedure. FIG. 9, comprising component FIGS. 9A and 9B

shows an alternative modification that can be mani foldly made to the FIG. 1 neural net layer to give it

10

15

20

25

30

35

45

50

55

60

65

16 training capability. This alternative modification seeks to avoid the need for a high-resolution multiplier MULT and complex addressing during back-propaga tion calculations in order that training can be imple mented. A respective up/down counter UDC is used instead of each word storage element WSE Correc tion of the word stored in counter UDCi is done a count at a time; and the counter preferably has at least one higher resolution stage in addition to those used to control the capacitances of digital capacitors DCii, DC(i+M), DC,0+N) and DC(i+M),0+N). Each up/down counter UDC has a respective counter control circuit CONiassociated therewith. Each counter control cir cuit CONimay, as shown in FIG. 9a, and described in detail further on in this specification simply consist of an exclusive-OR gate XORi. A row sign detector RSD detects whether the polar

ity of 8 is positive or negative, indicative of whether a row of weights should in general be decremented or incremented, and broadcasts its detection result via a row sign line RSL to all counter control circuits (CO Nii for i=1,..., M) in the row jassociated with that row sign detector RSD. Before making a back-propa gation calculation, a respective column sign detector CSD detects whether the polarity of x is positive or negative for each columnar position along the row which is to be updated, to provide an indication of whether it is likely the associated weight should be decremented or incremented. This indication is stored temporarily in a (column) sample and hold circuit CSH. Each column sample and hold circuit CSH is connected to broadcast its estimate via a column sign line CSLito all counter control circuits (CON for j=1, . . . N) in the columni associated with that sample and hold circuit CSH. Responsive to these indications from sign detectors CSDi and RSD, each respective counter control circuit CON decides in which direction up/- down counter UDCi will count to adjust the weight control signals Dij and Distored therein The counter control circuitry CONishould respond

to the sign of +6, being positive, indicating the response vi to be too positive, to decrease the capacitance to output line OL that is associated with the signal xi or -xi that is positive and to increase the capacitance to output line OL that is associated with the signal -xior xi that is negative, for each value of i. The counter con trol circuitry CONishould respond to the sign of +8, being negative, indicating the response v to be too nega tive, to increase the capacitance to output line OL that is associated with the signal -xi or xi that is negative and to decrease the capacitance to output line OL that is associated with the signal xi or -xt that is positive. Accordingly, counter control circuitry CON may simply consist of a respective exclusive-OR gate XOR as shown in FIG. 9a, if the following presumptions are valid. Each of the digital capacitors DC and

DC(i+M),0+N) is presumed to increase or decrease its capacitance as Di is increased or decreased respec tively. Each of the digital capacitors DC(i+M) and DC,0+N) is presumed to increase or decrease its capaci tance as Dii is increased or decreased respectively. A ZERO applied as up/down signal to up/down counter UDCi is presumed to cause counting down for Diand counting up for Dil. A ONE applied as up/down signal to up/down counter UDC is presumed to cause count ing up for Dii and counting down for Dii. Column sign detector CSD output indication is presumed to be a

5,146,542 17

ZERO when x is not negative and to be a ONE when xiis negative. Row sign detector RSD output indication is presumed to be a ZERO when 8 is not negative and to be a ONE when 8 is negative. Since the condition where xi or 8 is zero-valued is treated as if the zero- 5 valued number were positive, forcing a false correction which is in fact not necessary, and thus usually creating the need for a counter correction in the next cycle of back-propagation training, there is dither in the correc tion loops. However, the extra stage or stages of resolu tion in each up/down counter UDCi prevent high resolution dither in the feedback correction loop affect ing the capacitances of DC, DC(i+M), DC,0+N) and DC(i+ M),0+N). Analog multiplier AM can be readily modified to

develop its product output in balanced form. The Bult and Wallinga analog multiplier initially develops its product in balanced form and follows this with a push pull to single-ended converter connection using current mirror multipliers. An additional balanced to single ended converter can convert the push-pull product to opposite-sense output signal. With an analog multiplier generating balanced product signals, the +8, and -8 terms can be supplied to a voltage comparator that serves as the row sign detector RSD Alternatively, since the derivative y'always has the same sign (nor mally a positive one), one can use a voltage comparator to compare the voltages supplied to the asterisked and double-asterisked input terminals of the analog multi plier AM, for providing the row sign detector RSD. FIG. 10 shows the construction of counter UDC

being one that has a plurality of binary counter stages BCS1, BCS2, BCS3 that provide increasingly more sig nificant bits of the weight control signal Dii and of its one's complement Di FIG. 11 shows the logic within each binary counter stage which is implemented with MOS circuitry that is conventional in the art. FIGS. 10 and 11 make it clear that the opposite directions of counting for Dijand Dijcan be controlled responsive to a ZERO or ONE up/down control signal in either of 40 two ways, depending on whether Di is taken from Q outputs of the flip-flops and Dii is taken from their Q outputs, as shown, or whether Dij is taken from the Q outputs of the flip-flops and Dii is taken from their Q outputs. If the latter choice had been made instead, each 45 counter control circuit CON would have to consist of a respective exclusive-NOR circuit, or alternatively the CSD; and RSD, sign detectors would have to be of opposite logic types, rather than of the sane logic type.

FIG. 12 shows a further modification that can be 50 made to the FIG. 1 neural net layer modified per FIG. 9. Input driver amplifier ID is time-division-multi plexed in duplex circuitry DPXi for performing the function of the differential charge-sensing amplifier DQS in FIG. 9a. Only one set of interconnecting lines is used between neural net layers, both for conducting x--VBIAS signals to a neural net layer during forward propagation when dip is ZERO and for conducting Ai--VBIAS signals from that neural net layer during back-propagation when dip is ONE. An input multi- 60 plexer IMIt controlled from the dp, which is signal on the mode control line MCL, performs the time-division multiplexing for the neural net layer including input driver amplifier ID. An output multiplexer OM for the preceding neural net layer controlled from the dipsignal in its mode controlling MCL' selects yi-VBIAs to the interconnecting line between the layers during forward propagation when dip is ZERO, and selects Ai-i-VBIAS

10

15

20

25

30

35

55

65

18 from that interconnecting line during back-propagation when dip is ONE.

Similar further modification can be made to the FIG. 1 neural net layer modified per FIG. 4. In such case column sign detector CSD column sample and hold circuit CSHi, and column sign line CSLi would, of course, not be used. The time-division-multiplexed use of the input driver

amplifiers ID'saves integrated-circuit die area. The conservation of interconnecting lines between succes sive neural net layers is of substantial importance, since pin-out limitations on interconnected integrated circuits having different neural net layers therein are less re strictive of neural net layer size. A neural net layer with twice as many inputs and outputs can be constructed within an integrated circuit having given pin-out limita tions. The multiplexers employed in various portions of the

circuits described above are customarily constructed of single-pole switch elements, each of which single-pole switch elements is conventionally a so-called "transmis sion gate' connection of one or more field effect transis tors in CMOS design. A suitable transmission gate is provided by the paralleled channels of a p-channel FET and an n-channel FET having oppositely swinging con trol voltages applied to their respective gate electrode to control the selective conduction of those paralleled channels. One skilled in the art and acquainted with the forego

ing specification will be able to design numerous vari ants of the preferred embodiments described therein, and this should be borne in mind when construing the following claims. What is claimed is: 1. A processor for providing a response to a plurality

of input signal voltages, said processor comprising: an output line on which a weighted summation of said

input signal voltages appears; an amplifier having an input port to which said output

line connects and having an output port at which said response is provided;

a plurality, 2M in number, of input lines respectively identified by consecutive ordinal numbers first through 2Mth, M being a positive integer;

a plurality, 2M in number, of capacitive elements, each exhibiting a respective capacitance including at least a respective stray capacitance component, each said capacitive element connecting a respec tive one of said input lines to said output line, and being identified by the same ordinal number as the input line it connects to said output line;

means for referring said output line to a reference voltage during dc-restoration periods interspersed among times of normal operation;

means for applying voltages associated with zero valued first through Mth input signal voltages to said first through Mihinput lines during said dc-res toration periods;

means for applying voltages associated with zero valued first through Mth input signal voltages to said (M-1)h through 2Mth input lines during said dc-restoration periods; and

means for applying M input signal voltages in bal anced form each to a respective pair of said input lines, identified by ordinal numbers M apart, so that said input lines in each respective pair simulta neously differ from the respective voltages applied to them during said dc-restoration periods by volt

5,146,542 19

ages of the same amplitude and opposite polarity; the difference in the capacitances of the capacitive elements connecting to said output line from said pair of input lines determining the weighting of the input signal applied in balanced form on that said 5 pair of input lines, as appears in said weighted sum mation of input signals.

2. A plurality, N in number, of processors of the type set forth in claim 1, respectively identified by respective consecutive ordinal numbers first through Nth, N being 10 a positive integer, said first through Nth processors hav ing each set of their input lines that are identified by the same ordinal number connected to be receptive of the same input signal, thereby to provide a neural network layer. 15

3. A processor as set forth in claim 1 wherein said amplifier is of a type that includes a charge-sensing amplifier for sensing the charge appearing on said out put line during said times of normal operation, which said charge-sensing amplifier includes: 20

a differential-input amplifier having an output termi nal, having an inverting input terminal to which said output line connects, and having a non-invert ing input terminal to which said reference voltage is applied; 25

an integrating capacitive element having a first plate connected to the inverting input terminal of said differential-input amplifier and having a second plate;

means for connecting the output terminal of said 30 differential-input amplifier to the second plate of said integrating capacitive element during said times of normal operation;

means for applying said reference voltage to the sec ond plate of said integrating capacitive element 35 during portions of said do-restoration periods; and

means for establishing a conductive path from the output terminal of said differential-input amplifier to the inverting input terminal of said differential input amplifier during portions of said dc-restora- 40 tion periods when said reference voltage is applied to the second plate of said integrating capacitive element,

4. A processor as set forth in claim 1 wherein the capacitances of each pair of said capacitive elements 45 having respective ordinal numbers different by M add together to a same prescribed value.

5. A processor as set forth in claim 4 wherein each of said capacitive elements is a respective digital capacitor having a structure equivalent to the structure of each of 50 the others, each said digital capacitor having a capaci tance that is programmed by a respective binary num ber programming signal and that is altered responsive to a change in said respective binary number programming signal. 55

6. A processor for providing an "axon' response to a plurality of "synapse" input signal voltages, said proces sor comprising:

an output line on which a weighted summation of said input signal voltages appears; 60

a non-linear amplifier having an input port to which said output line connects and having an output port at which said "axon" response is provided, said non-linear amplifier exhibiting sigmoidal non linearity in its response characteristic said sigmoi- 65 dal response characteristic having a portion of response characteristic with maximum or close-to maximum sensitivity between portions of response

20 characteristic with substantially-less-than-max inum sensitivity;


a plurality, 2M in number, of capacitive elements, each exhibiting a respective capacitance including at least a respective stray capacitance component, each said capacitive element connecting a respec tive one of said input lines to said output line, and being identified by the same ordinal number as the input line it connects to said output line; and

means for applying M input signal voltages in bal anced form each to a respective pair of said input lines, identified by ordinal numbers M apart, the difference in the capacitances of the capacitive elements connecting to said output line from said pair of input lines determining the weighting of the input signal applied in balanced form on that said pair of input lines, as appears in said weighted sum mation of input signals, whereby the effects of said stray capacitances on said weighted summation of said input signals is reduced-said processor in cluding the improvement wherein said means for applying M input signal voltages is of a type for applying said signal voltages in balanced form at levels such as to cause, responsive to certain combi nations of said M input signal voltages all within their ordinary ranges, said non-linear amplifier to operate in a portion of its sigmoidal response char acteristic having substantially-less-than-maximum sensitivity.

7. A processor for providing an "axon' response to a plurality of "synapse' input signal voltages, said proces sor comprising:

an output line on which a weighted summation of said input signal voltages appears;

a plurality, 2M in number, of input lines respectively identified by consecutive ordinal numbers first through 2Mh, M being a positive integer;

a plurality, 2M in number, of capacitive elements, each exhibiting a respective capacitance including at least a respective stray capacitance component, each said capacitive element connecting a respec tive one of said input lines to said output line, and being identified by the same ordinal number as the input line it connects to said output line;

means for referring said output line to a reference voltage during dc-restoration periods interpersed among times of normal operation;

means for applying voltages associated with zero valued first through Mih input signal voltages to said first through Mihinput lines during said dc-res toration periods;

means for applying voltages associated with zero valued first through Mih input signal voltages to said (M-1)th through 2Mth input lines during said dc-restoration periods;

means for applying M input signal voltages in bal anced form each to a respective pair of said input lines, identified by ordinal numbers M apart, the difference in the capacitances of the capacitive elements connecting to said output line from said pair of input lines determining the weighting of the input signal applied in balanced form on that said pair of input lines, as appears in said weighted sum mation of input signals, whereby the effects of said

5,146,542 21

stray capacitances on said weighted summation of said input signals is reduced;

a charge-sensing amplifier having an input port to which said output line connects and having an

22 sigmoidal response characteristic having substan tially-less-than-maximum sensitivity.

9. A processor for providing an "axon' response to a plurality of "synapse' input signal voltages, said proces

output port, said charge-sensing amplifier being of 5 sor comprising: a type maintaining a prescribed value of voltage on said output line irrespective of the value of said weighted summation of said input signals; and

means for providing, as said "axon' response, a non linear voltage response to voltage appearing at the 10 output port of said charge-sensing amplifier.

8. A neural network having a plurality, L in number, of neural net layers respectively identified by consecu tive ordinal numbers zeroeth through (L-1)th, L being a positive integer at least two, each said neural net layer 15 including: a respective plurality, 2M in number, of input lines

respectively identified by consecutive ordinal num bers first through 2Mth, the respective M for each neural net layer being a positive integer at least 20 two;

means for applying M input signal voltages in bal anced form each to a respective pair of said input lines, identified by ordinal numbers M apart; and

a respective plurality, N in number, of processors 25 respectively identified by respective consecutive ordinal numbers first through Nth, the respective N for each neural net layer being a positive integer at least two, each said processor comprising:

a respective output line; - 30 a respective non-linear amplifier having an input port

to which said respective output line connects and having an output port at which a respective output signal is provided, the non-linear amplifiers of the processors in each of said neural net layers after the 35 Zeroeth supplying their output signals to determine the input signal voltages of respective processors in neural net layers of lower ordinal numbering, each said non-linear amplifier exhibiting a sigmoidal non-linear response characteristic having a portion 40 of response characteristic with maximum or close to-maximum sensitivity between portions of re sponse characteristic with substantially-less-than maximum sensitivity; and

a plurality, 2N in number, of capacitive elements, 45 each exhibiting a respective capacitance including at least a respective stray capacitance component, each said capacitive element connecting to said output line a respective one of said input lines of that said neural net layer as includes that said pro- 50 cessor, and being identified by the same ordinal number as the input line it connects to said respec tive output line, the difference in the capacitances of the capacitive elements connecting to said out put line from each said pair of input lines determin- 55 ing the weighting of the input signal applied in balanced form on that said pair of input lines as appears on said output line-said neural network including the improvement wherein in each said neural net layer said means for applying M input 60 signal voltages in balanced form each to a respec tive pair of said input lines includes:

means for applying said signal voltages in balanced form at levels such as to cause, responsive to cer

an output line on which a weighted summation of said input signal voltages appears;

a non-linear amplifier having an input port to which said output line connects and having an output port from which a non-linear response to said weighted summation of said input signal voltages is provided, which non-linear response corresponds to said "axon' response;


a plurality, 2M in number, of capacitive elements, each of said capacitive elements being a respective digital capacitor of the same type as each of the others, each said digital capacitor having a respec tive capacitance that is programmed by a respec tive binary number programming signal, its respec tive capacitance including at least a respective stray capacitance component, each said capacitive element connecting a respective one of said input lines to said output line and being identified by the same ordinal number as the input line it connects to said output line, the capacitances of each pair of said capacitive elements having respective ordinal numbers different by Madding together to a same prescribed value;

means for applying M"synapse" input signal voltages in balanced form each to a respective pair of said input lines, identified by ordinal numbers M apart, the difference in the capacitances of the capacitive elements connecting to said output line from said pair of input lines determining the weighting of the input signal applied in balanced form on that said pair of input lines, as appears in said weighted sum mation of input signals, whereby the effects of said stray capacitances on said weighted summation of said input signals is reduced; and

means for applying as programming signals to said (M-1)th through 2Mth capacitive elements binary numbers that are the one's complements of binary numbers applied as programming signals to said first through Mth capacitive elements respectively.

10. A processor as set forth in claim 9 having associ ated therewith:

a respective binary counter for supplying the pro gramming signal for each of said first through Mh capacitive elements, which binary counter is identi fied by the same ordinal number as the capacitive element it supplies programming signal; and

means for controlling the incrementing and decre menting of the binary count in each of said binary counters.

11. A processor as set forth in claim 9 including: a respective memory array for binary numbers hav

ing a respective word storage element for each of said first through Mth capacitive elements for sup plying the binary number applied to that capacitive element as its programming signal.

12. A plurality, N in number, of processors of the tain combinations of said M input signal voltages 65 type set forth in claim 11, respectively identified by all within their ordinary ranges, certain ones of said non-linear amplifiers in said processors in said neu ral net layer each to operate in a portion of its

respective consecutive ordinal numbers first through Nth, N being a positive integer, said first through Nth processors having each set of their input lines that are

5,146,542 23

identified by the same ordinal number connected to be receptive of the same input signal, thereby to provide a neural network layer.

13. A plurality, L in number, of neural net layers respectively identified by consecutive ordinal numbers zeroeth through (L-1)th, L being a positive integer, each of said zeroeth through (L-1)th neural net layers comprising a respective plurality, M in number, of pro cessors of the type set forth in claim 11, respectively identified by respective consecutive ordinal numbers first through Mth, said first through Mth processors having each set of their input lines that are identified by the same ordinal number connected to be receptive of the same input signal, the non-linear amplifiers of said first through Mth processors in each of said neural net layers after the zeroeth supplying their output signals to determine the input voltage signals identified by corre sponding ordinal numbers of respective processors in said neural network layer of next lower numbering.

14. A neural net layer comprising a respective plural ity, M in number, of processors of a type as set forth in claim 11, respectively identified by respective consecu tive ordinal numbers first through Mth, said first through Mih processors having each set of their input lines that are identified by the same ordinal number connected to be receptive of the same input signal, said neural net layer being of a type for being trained during at least one training period of time, each followed by a respective operating period of time, said neural net layer having

differentiation means associated with each said pro cessor for determining the slope of non-linear re sponse from its said non-linear amplifier for a pre determined set of said input signal voltage signals;

means for multiplying the slope of non-linear re sponse for the respective non-linear amplifier of each said processor by an error signal associated with that processor thereby to generate a respec tive modified error signal associated with each processor;

means for applying, during said training time, said modified error signal associated with each proces sor to said output line associated with that said processor;

means for changing the binary number stored in each said word storage element during each training period by an amount related to the product of (1) the input signal applied to the one of the first through Mihinput lines identified by the same ordi nal number as the one of said capacitive elements receiving its programming signal from that word storage element and (2) the modified error signal applied to the output line to which that one of said capacitive elements connects; and

a plurality Min number of differential charge-sensing amplifiers being respectively identified by consecu tive ordinal numbers first through Mth, having respective first input ports respectively connected to the input lines identified by the same ordinal number as their own, having respective second input ports respectively connected to the input lines identified by the ordinal number M higher than their own, and having respective output ports responding to the difference of charge flow to their respective said first and second input ports during each said training period of time for generating an error signal identified by the same ordinal number as said processor generating it.

O

15

20

25

35

40

45

SO

55

65

24 15. A plurality, L in number, of neural net layers of

the type set forth in claim 14, respectively identified by consecutive ordinal numbers zeroeth through (L-1)th, L being a positive integer, the output ports of the pro cessors in each of said neural net layers other than the zeroeth supplying the output signals from their non-lin ear amplifiers to determine the input voltage signals of respective processors in neural network layers of next lower ordinal numbering, and in each neural network layer other than the zeroeth, each processor therein being provided with the correspondingly numbered error signal from the neural net with next lower ordinal number.

16. A neural net layer as set forth in claim 14 wherein each said word storage element comprises:

a respective up/down binary counter having respec tive first and second count output ports from which the complementary binary numbers stored therein are available, having a respective clock input port connected to receive a repetitive clocking signal, and having a respective up/down control port; and

a respective exclusive-OR gate being associated with each said up/down binary counter, having an out put port connecting to the up/down control port of said up/down binary counter associated therewith, having a first input port receptive of a signal corre sponding to the sign of input signal applied in bal anced form to the pair of input lines to which con nect the capacitive elements receiving their pro gramming signals responsive to the binary numbers stored within said associated up/down counter, and having a second input port receptive of a signal corresponding to the sign of the modified error signal applied to the output line to which connect the capacitive elements receiving their program ming signals responsive to the binary numbers stored within said associated up/down counter.

17. A neural net, for responding to a plurality M in number of input signal voltages identified by consecu tive ordinal numbers first through Mih, to generate a plurality N in number of output signal voltages identi fied by consecutive ordinal numbers first through Nth, M and Neach being a respective positive plural integer, said neural net comprising:

a plurality 2M in number of input lines respectively identified by consecutive ordinal numbers first through 2Mth;

a plurality N in number of output lines respectively identified by consecutive ordinal numbers first through Nth;

a respective capacitive element for connecting each of said first through 2Mth input lines to each of certain said output lines;

a respective input driver amplifier responding to each input signal voltage for applying non-inverted re sponse to that input signal to the one of said first through Mihinput lines identified by the same ordi nal number as that input signal voltage and for applying inverted response to that input signal voltage to the one of said (M-1)th through 2Mth input lines identified by an ordinal number M above the ordinal number identifying that input signal voltage;

a respective output driver amplifier responding to the change in charge on each said output line for gen erating a respective said output signal voltag, each said output driver amplifier having a non-linear voltage versus charge transfer characteristic;

5,146,542 25

a respective differentiator associated with each out put driver amplifier for measuring the slope of its transfer characteristic for a given set of input volt age signals;

means for temporarily storing the measured slopes of 5 the transfer characteristic of each of said output driver amplifiers;

a respective analog multiplier associated with each said output driver amplifier, each said analog multi plier multiplying the temporarily stored measured slope of the transfer characteristic of the one of said output driver amplifiers with which it is associated

10

26 said bridge configurations

2,737. v. - patentimages.storage.googleapis.com · "analog mos integrated circuits for signal...

Documents