hopfield rapor
TRANSCRIPT
-
8/3/2019 Hopfield rapor
1/16
-
8/3/2019 Hopfield rapor
2/16
1 INTRODUCTIONJohn J. Hopfield published two papers about an associative model that is one of the
milestones in the field of neural networks. This neural network needs no synchronization and
it is very successful at optimization and pattern recognition.Many of ideas in these papers were based on the previous work of other researchers.
These researchers had considered more general associative memory models in the 1970s, but
by restricting the architecture of the network to a symmetric connection matrix with a zero
diagonal, it was possible to design recurrent networks with stable states.
In addition to this there are several other reasons why Hopfields papers have had such
an impact. First, he identified a close analogy between his neural network and the Ising model
of magnetic material. Hopfield also had close contacts with VLSI chip designers, because of
the bell laboratories. The fact that Hopfield addressed the implementation issues of his
network distinguished him from most previous neural network researchers.
1.1 Terminology:y Symmetric connections: if the weight between neurons doesnt change it is asymmetric connection.
y Asymmetric connections: if the weight between neurons doesnt change it is asymmetric connection.
Symmetric and Asymmetric Connections
y Asynchronous: when firing a neuron, first calculates the weighted input sum andupdates immediately. This can be done in a fixed order, or neurons can be picked at random,
which is called asynchronous random updating.y Synchronous: the weighted input sums of all neurons are calculated without updatingthe neurons. Then all neurons are set to their new value, according to the value of their
weighted input sum.
y Associative memory: Recalling a previously experienced item by thinking ofsomething that is linked with it, thus invoking the association. For example when we see red
-
8/3/2019 Hopfield rapor
3/16
light we understand it as stop and green as go. So colors are reminds us the commands.
This is a associative memory.
y Ising model: this physical system consists of particles that capable of adopting one oftwo states. In the case of ferromagnetic materials, their atoms can be modeled as particles of
spin 1/2 (up) or spin 1/2 (down). The spin points in the direction of the magnetic field. All
tiny magnets interact with each other. This causes some of the atoms to flip their spin until
equilibrium is reached and the total magnetization of the material reaches a constant level,
which is the sum of the individual spins.
Ising model
2 HOPFELD NEURAL NETWROK2.1 A physical analogy with memory
To show the similarity between the energy and the memory, we can think a bowl in
which a ball bearing is allowed to roll freely as shown in figure below. Suppose we let the ball
go from a point somewhere up the side of the bowl. The ball will oscillates around bottom of
the bowl and after some oscillations ball will be rest at the bottom. We can describe what
happened, in terms of energy. At first, system have a potential energy according to its height,
higher the ball the greater its energy.
-
8/3/2019 Hopfield rapor
4/16
Eventually the ball comes to rest at the bottom, energy is now at a minimum since any
other location of the ball is associated with some potential energy, which may be lost on
allowing the bowl to reach equilibrium. This equilibrium state doesnt change with the
starting point. There is another way of thinking about this process that ties in with our ideas
about memory. This example illustrated to understand the relation. Thus, we suppose that the
ball comes to rest in the same place each time because it remembers where the bottom of
the bowl is.
If we now use a corrugated surface instead of a single depression we may store many
memories. If the ball is now started somewhere on this surface, it will eventually come to
rest at the local depression that is closest to its initial starting point. That is, it evokes the
stored pattern which is closest to its initial partial pattern or cue. This corresponds to an
energy minimum of the system. The memories shown correspond to states x1, x2, x3 where
each of these is a vector.
There are therefore two complementary ways of looking at what is happening. One is to
say that the system falls into an energy minimum; the other is that it stores a set of patterns
and recalls that which is closest to its initial state.
2.2 Hopfield networkNow we are going to apply principle we talk above, to the construction of a neural
network capable of performing associative recall. Consider the network consisting of three
nodes shown below. Each node is connected to every other node and the connection weights
are symmetric. Thresholds are all assumed to be zero. Notice that the flow of information in
this type of net is not in a single direction, since it is possible for signals to flow from a node
back to itself via other nodes. We say there is feedback in the network or that it is recurrent
because nodes may be used repeatedly to process information.
-
8/3/2019 Hopfield rapor
5/16
The state of the network at any time is given as (x1, x2, x3). Suppose we now start this
net in some initial state, choose a node at random and let it update its output or fire. That is,
the chosen node evaluates its activation in the normal way and outputs a 1 if this is greater
than or equal to zero and a 0 otherwise. The net now finds itself either in the same state as it
started in, or it changes one of the states. We now choose another node at random, let itupdate or fire, and repeat this many times. This process defines the dynamics of the net.
In describing these state transitions it is convenient to attach a numeric label to each
state and the most natural way of doing this is to interpret the Boolean state vector as a binary
number, so that state (x1, x2, x3) is labeled with 4x1+2x2+x3. For example, (1, 1, 0) is state 6.
For each network state, there are three possible outcomes for the next state depending on
which of the three nodes is chosen to fire. Suppose, for example, the net starts in state (1, 0, 1)
(label 5) and node 1 fires. The activation of this node is given by
a=w13x3+w12x2=21+10=2.Then, since this is less than 0, the new output is also 0 and the new state is (0, 0, 1)
(label 1); in summary, state 5 goes to state 1 when node 1 fires. Repeating this working for
nodes 2 and 3 firing, the new states are 7 and 4 respectively. By working through all initial
states and node selections it is possible to evaluate every state transition of the net as shown
as;
Initial State After A Node Firingx1 x2 x3 Label # 1 2 3
0 0 0 0 4 2 1
0 0 1 1 1 3 1
0 1 0 2 6 2 3
0 1 1 3 3 3 3
1 0 0 4 4 6 4
1 0 1 5 1 7 4
1 1 0 6 6 6 6
1 1 1 7 3 7 6
-
8/3/2019 Hopfield rapor
6/16
Notice that a state may or may not change when a node fires. This information may also
be represented in graphical form as a state transition diagram, shown in figure below;
For example, starting in state 5 we see from the diagram that there is an equal
probability of 1/3 of going to states 1, 7, or 4, which is reflected in the three arcs emanating
from state 5 in the diagram. Again, starting in state 1 and updating nodes 1 or 3 results in no
state change, so there is a probability of 2/3 that state 1 stays as it is; however, choosing node
2 to update results in a transition to state 3. The no-change condition gives an arc that starts
and ends at the same state.
Another firing method is that; classify the units of a network according to their state: the
first set contains the units with state 1, the second set the units with state 1. There are edges
linking every unit with all the others, so that some edges go from one set to the other. We now
randomly select one of the units and compute its attraction by the units in its own set and
the attraction by the units in the other set. The attraction is the sum of the weights of all
edges between a unit and the units in its set or in the other one. If the attraction from the
outside is greater than the attraction from its own set, the unit changes sides by altering its
state. If the external attraction is lower than the internal, the unit keeps its current state.[74]
-
8/3/2019 Hopfield rapor
7/16
2.3 Defining an energy for the networkThe dynamics of the net are described completely by the state transition table or
diagram. It has demonstrated the existence of stable states. On the other hand energy basedform is more accurate to find stable states. We can think every node effect other with the
associated weight. The use of the energy concept in this context may be introduced using an
analogy with a simple physical system. Thus, consider a pair of objects joined by a spring, use
of a tension spring, which tends to pull the objects together. Since work has to be done to
draw the objects apart, the energy minimum of this system occurs when the objects are close
to each other or, in other words, tend towards the same position in space. Compression spring
energy minimum occurs when the objects are far apart.
Now consider two nodes i, j in a Hopfield net connected by a positive weight +w, as
shown above. We claim that positive and negative weights are analogous to the tension and
compression springs respectively, since positive weights tend to make the nodes take on the
same output values while negative weights tend to force different values.
The constraint description may now be set within the energy framework by assigning
high energies to states that tend to get destabilized and low energies to those that are
reinforced. One way of doing this is to define the internode energy eij by The energy of the whole network E is found by summing over all pairs of nodes
-
8/3/2019 Hopfield rapor
8/16
Since the connections are symmetric, the sum includes each pair twice so we put the
coefficient. When we adding the threshold value for every node we obtain the energy formula
for Hopfield neural network:
2.4 Finding the weightsp to now nothing has been said about how the network can be made to store patterns
from a training set. In his original paper, Hopfield (1982) did not give a method for training
the networks under an incremental, iterative process. But commonly to store pattern into the
network biologically inspired Hebb training method is used.
The main purpose of the Hopfield neural network is constraint the outputs of the
network according to given weights. With this aspect, we can adjust our weights with therelations between them. For example, two nodes that tend, on average, to take on the same
value over the training set so that the pairs 0, 0 or 1, 1 dominates; we say that the nodes are
correlated. The pair 1, 1 will be reinforced by there being a positive internode weight. If, on
the other hand, the two nodes tend, on average, to take on opposite values with pairs 0, 1 or 1,
0, then this will be reinforced by a negative internode weight.
2.4.1 The Hebb ruleIn 1949, neuropsychologist D. O. Hebb describes a possible way for biological neurons
learn. And he said that:
When an axon of cell A is near enough to excite a cell B and repeatedly or persistently
takes part in firing it, some growth process or metabolic change takes place in one or
both cells such that As efficiency, as one of the cells firing B, is increased.
This description can be adapted to our training algorithm in three steps:
1. Choose a pattern from the training set at random.2. Present the components of this pattern at the outputs of the corresponding nodes of the
net.
3. If two nodes have the same value then makes positive increment to the internodeweight. If they have opposite values then make negative decrement to the weight.
Iteration of these three steps many times constitutes a training algorithm whose learning rule
may be written mathematically as;
-
8/3/2019 Hopfield rapor
9/16
-
8/3/2019 Hopfield rapor
10/16
This expression can also be written as
States are binary so xi = xi
2;
Our Hopfield energy function is;
And we can make some arrangements to our energy function to assimilate it to Hopfield
energy function;
For every column we find the weight as a constant -2 and threshold value -1.
The minimum of the function corresponds to the situation in which just one rook has
been positioned in every column. Similarly, for the rows of the board we define the function
E2 according to
-
8/3/2019 Hopfield rapor
11/16
-
8/3/2019 Hopfield rapor
12/16
4 APPLICATIONS OF HOPFIELD NETWORKSAs I mentioned before, when Hopfield was working for Bell industries, he proposed an
electrical realization of his model. He used amplifiers for Hopfield units and obtains
complements of these units with inverters and fed back these to the inputs. Symbolic
representation of this circuit is shown in figure. In this figure circles in contacts, are
represented resistance and Hopfield adjust the weights with these resistance.
-
8/3/2019 Hopfield rapor
13/16
For example, contains a resistor with resistance r13 = 1/w13. The constants wij
represent the weights of the Hopfield network between the units i and j. Inhibiting
connections between one unit and another (that is, connections with negative weights) are
simulated by connecting one inverted output of a unit to the other one.
One of the most important implementation of Hopfield network is realized by optical
way. The networks binary values are projected through the vertical lens to the left of the
arrangement. The lens projects each value xi onto the corresponding row of an optical mask.
Each row i in the mask is divided into fields which represent the n weights wi1, wi2 . . . win. on
the mask, reflected light emitting or reflecting according to attended weight. The individual
unit states are projected using light emitting diodes. The light going through the mask is
collected by another lens and obtained a single value. The amount of light that goes through
the mask is proportional to the product of x i and wij at each position ij of the mask.
In literature Hopfield neural network used for two purposes one of them is pattern
recognition and the other is optimization problems. In pattern recognition, binary images are
sampled and for each field a node will be attend. When image is stored to the network weights
will be updated. For more patterns node number will be increase exponentially.
For optimization problems can be written in an analytical form isomorphic to the
Hopfield energy function, it can be solved by a Hopfield network. We can assume that every
unit in the network is simulated by a small processor. The states of the units can be computed
asynchronously by transmitting the current unit states from processor to processor. There is
no need for expensive synchronization and the task is solved by a massively parallel system.
This strategy can be applied to all those combinatorial problems for whose solution large
mainframes have traditionally been used.
-
8/3/2019 Hopfield rapor
14/16
4.1 HNN in image processingBecause of the usage of associative memory Hopfield neural network (HNN) finds
largely working area in image processing. In many studies Hopfield NN was used for image
recognition such as hand writing recognition, face and gender recognition with genetic
algorithm, edge detection, image segmentation, image restoration and image compression.
In one study different characters are stored in HNN and when input is degraded and
noise image HNN is enhance the image(it recognize the original image from the degraded
image). Results are very successful with less recognized characters and under a fifty persent
noise.
As seen in the results HNN is very successful in image recognition. But we shouldnt
forget only binary images can be an input for Hopfield neural network.
4.2 Optimization problemsAs I mentioned before, if we can be write the given problem in an analytical form
isomorphic to the Hopfield energy function, it can be solved by a Hopfield network. Hopfield
and Tank were the first to try to solve the TSP using a neural network. But such a complex
problem solved by HNN with binary neurons it strict a local minimum. So in 1984 Hopfield
replaced binary neurons by neurons whose instantaneous activity increases with input and
applied this to the TSP.
-
8/3/2019 Hopfield rapor
15/16
Results of the travelling sales man problem are shown above. In these figures first and
second are continuous Hopfield model and the last one is the binary Hopfield model. As its
clearly shown in the figure continuous HNN finds the global minimum but the binary HNN
finds the local minimum.
Heuristics have been proposed for the TSP, starting with the classical work by
Kernighan and Lin. One of the first to deal with limits of the Hopfield model with constant
size for the solution of the TSP was Abu-Mostafa. Although this almost meant the death of
the traveling salesman, the Hopfield model and its stochastic variants have been applied in
many other fields, such as psychology, simulation of ensembles of biological neural networks,
and chaotic behavior of neural circuits.
5 CONCLUSIONHopfield NN is one of the most important milestones of recurrent networks. Some
properties of this network is:
A recurrent network with all nodes connected to all other nodes Nodes have binary outputs (either 0,1 or -1,1) Weights between the nodes are symmetric wij = wji No connection from a node to itself is allowed Nodes are updated asynchronously (i.e. nodes are selected at random) The network has no hidden nodes or layer
Main advantages of this system are, do not need any kind of synchronization; they
guarantee that a local minimum of the energy function will be reached. In pattern recognition
HNN is very successful but with the increasing of the stored pattern number the network
expands and the success of the network is decrease.
-
8/3/2019 Hopfield rapor
16/16
In simple optimization problems HNN with binary neurons can be used but it only finds
local minimum. For more complex optimization problems like travelling salesman or rout
planning we must use HNN with continuous neuron.
There are a lot of areas that Hopfield used with different algorithms. But I think
Hopfield can be very successful in classification studies. The studies in this task are much
related and can be developed.
1) Hopfield J., 1982, Neural networks and physical systems with emergent collectivecomputational abilities, Proc. Natl. Acad. Sci. USA
2) Rojas, R., 1996, Neural Networks A System Introduction, Springer-Verlag Berlin3) Gurney, K., 1997, An Introduction to Neural Networks ,UCL Press, London4) Hopfield J., 1985, Neural Computation of Decisions in Optimization Problems,
Biological Cybernetics, Springer-Verlag
5) Boden M., 2001, AGuide to Recent Neural Networks and Backpropagation ,Halmstad University