ie 585
DESCRIPTION
IE 585. Competitive Network – I Hamming Net & Self-Organizing Map. Competitive Nets. Unsupervised MAXNET Hamming Net Mexican Hat Net Self-Organizing Map (SOM) Adaptive Resonance Theory (ART) Supervised Learning Vector Quantization (LVQ) Counterpropagation. Clustering Net. - PowerPoint PPT PresentationTRANSCRIPT
IE 585
Competitive Network – I
Hamming Net & Self-Organizing Map
2
Competitive Nets
Unsupervised• MAXNET• Hamming Net• Mexican Hat Net• Self-Organizing Map (SOM)• Adaptive Resonance Theory (ART)
Supervised• Learning Vector Quantization (LVQ)• Counterpropagation
3
Clustering Net
• Number of input neurons equal to the dimension of input vectors
• Each output neuron represents a cluster
the number of output neurons limits the number of clusters that can be performed
• The weight vector for an output neuron serves as a representative for the input patterns which the net has placed on that cluster
• The weight vector for the winning neuron is adjusted
4
Winner-Take-All
• The squared Euclidean distance is used to determine the closest weight vector to a pattern vector
• Only the neuron with the smallest Euclidean distance from the input vector is allowed to update
5
MAXNET
• Developed by Lippmann, 1987• Can be used as a subset to pick the node
whose input is the largest• Completely interconnected (including self-
connection)• Symmetric weights• No training• Weights are fixed
6
Architecture of MAXNET
1
1
1
1
/1
/1
/1
/1
/1
/1
ji
jiwij if /1
if 1
otherwise 0
0 if )(
FunctionTransfer
xxxf
7
Procedure of MAXNET
Initialize activations and weights
Update activation of each node
If more than one node has a nonzero activation, continue; otherwise, stop
newj
oldj
j
oldjij
newj
xx
xwfx
)(
8
Hamming Net
• Developed by Lippmann, 1987• A maximum likelihood classifier• used to determine which of several exemplar
vectors is most similar to an input vector• Exemplar vectors determine the weights of the
net• Measure of similarity between the input vector
and the stored exemplar vectors is (n – HD between the vectors)
9
Weights and Transfer Function of Hamming Net
in vector s0' of # 0 if
1 if
1
1
(0,1)binary 2
2
1
(-1,1)bipolar
b
i
iij
b
iij
wx
xw
nw
xw
otherwise 0
0 if )(
Function)(Identity
FunctionTransfer
xxxf
10
Architecture of Hamming Net
x1 x2 x3 x4
B B
y1 y2
MAXNET
11
Procedure of the Hamming Net
Initialize weights to store the m exemplar vectors
For each input vector x
compute
initialize activation for MAXNET
MAXNET iterates for find the best match exemplar
i
iijjY xwbnetj
jYj nety )0(
12
Hamming Net Example
13
Mexican Hat Net
• Developed by Kohonen, 1989• Positive weight with “cooperative neighboring”
neurons• Negative weight with “competitive neighboring”
neurons• Not connect with far away neurons
14
Teuvo Kohonen
• http://www.cis.hut.fi/teuvo/ (his own home page)
• published his work starting in 1984
• LVQ - learning vectorquantization
• SOM - self organizingmap
Professor atHelsinki Univ.Finland
15
SOM• Also called Topology-Preserving Maps or Self-Organizing
Feature Maps (SOFM)• “Winner Take All” learning (also called competitive learning)• winner has the minimum Euclidean distance• learning only takes place for winner• final weights are at the centroids of each cluster• Continuous inputs, continuous or 0/1 (winner take all)
outputs• No bias, fully connected• used for data mining and exploration• supervised version exists
16
Architecture of SOM Net
I
N
P
U
T
S
(a’s)
O
U
T
P
U
T
S
(y’s)
Input
Layer
W
n
17
Kohonen Learning Rule Derivation
2222 min wwaawa winner
w)a-(w
wa
w22 )(
E2
training.during decreasesusually and
7.01.0
)(
waw
18
Kohonen Learning
oldj
oldj
oldj
oldj
newj
wa
waw
www
.
..
..
)1(
][
19
Procedure of SOM
Initialize weights uniformly and normalize to unit lengthNormalize inputs to unit length Present an input vector x
calculate Euclidean distance between x and all Kohonen neuronsselect winning output neuron j (with the smallest distance)update the winning neuron
re-normalize weights to j (sometimes skipped)present next training vector
][ oldoldnew waww
20
Method
Normalize input vectors, a, by:
Normalize weight vectors, w, by:
Calculate distance from a to each w by:
iki
kiki
a
aa
2
iji
jiji
w
ww
2
i
jikij wad 2
21
Min d wins (this is the winning neuron)Update w of the min d neuron by:
Return to 2 and repeat for all input vectors a
Reduce if applicableRepeat until weights converge (stop
changing)
jioldkioldnew wawwjiji
22
SOM Example - 4 patterns
p1 1 1p2 1 0p3 0 1p4 0 0neuron1 neuron2 neuron3 neuron4 dist1 dist2 dist3 dist4w1 w2 w1 w2 w1 w2 w1 w2
0.2 0.3 0.5 0.4 0.1 0.7 0.5 0.6 1.13 0.61 0.9 0.410.2 0.3 0.5 0.4 0.1 0.7 0.625 0.7 0.73 0.41 1.3 0.6306250.2 0.3 0.625 0.3 0.1 0.7 0.625 0.7 0.53 0.880625 0.1 0.4806250.2 0.3 0.625 0.3 0.075 0.775 0.625 0.7 0.13 0.480625 0.60625 0.880625
0.15 0.225 0.625 0.3 0.075 0.775 0.625 0.7 1.323125 0.630625 0.90625 0.2306250.15 0.225 0.625 0.3 0.075 0.775 0.71875 0.775 0.773125 0.230625 1.45625 0.6797270.15 0.225 0.71875 0.225 0.075 0.775 0.71875 0.775 0.623125 1.117227 0.05625 0.5672270.15 0.225 0.71875 0.225 0.05625 0.83125 0.71875 0.775 0.073125 0.567227 0.694141 1.117227
0.1125 0.16875 0.71875 0.225 0.05625 0.83125 0.71875 0.775 1.478633 0.679727 0.919141 0.1297270.1125 0.16875 0.71875 0.225 0.05625 0.83125 0.789063 0.83125 0.816133 0.129727 1.581641 0.7354710.1125 0.16875 0.789063 0.16875 0.05625 0.83125 0.789063 0.83125 0.703633 1.313596 0.031641 0.6510960.1125 0.16875 0.789063 0.16875 0.042188 0.873438 0.789063 0.83125 0.041133 0.651096 0.764673 1.313596
0.084375 0.126563 0.789063 0.16875 0.042188 0.873438 0.789063 0.83125
=0.25
23
Movement of 4 weight clusters
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Neuron 1 Neuron 2
Neuron 3 Neuron 4
24
Adding a “conscience”
• prevents neurons from winning too many training vectors using a bias (b) factor
• winner had min (d-b) where
bj=10(1/n-fj) (n=# output neurons)
fjnew=fjold+0.0001(yj-fjold)
finitial=1/n
• for neurons that win, b becomes negative and for neurons that don’t win, b becomes positive
25
Supervised Version
• Same, except if the winning neuron is “correct” use same weight update:
wnew = wold + (a - wold) and
• if winning neuron is “incorrect” use:
wnew = wold - (a - wold)