artificial neural networks ch14. objectives discuss networks that are very similar in structure and...
TRANSCRIPT
Artificial Neural Networks
Ch14
Objectives
Discuss networks that are very similar in structure and operation to Hamming network.
They use the associative learning rule to adaptivelyadaptively learn to classify pattern.
Three such networks are introduced in this chapter: the competitive network, the self-organizing feature map (SOFM) network and the learning vector quantization (LVQ) network.
Hamming Network
The first layer performs a correlation between the input vector and the prototype vector.
The second layer performs a competition to determine a winner which of the prototype vectors is closest to the input vector.
Layer 1: Correlation
The prototype vector: {p1, p2, …, pQ} The weight matrix and the bias vector for Layer 1:
The output of the first layer:These inner products indicatehow close each of the prototypepatterns is to the input pattern.
R
R
R
TQ
T
T
TS
T
T
12
1
2
1
1 , b
p
p
p
w
w
w
W
R
R
R
TQ
T
T
pp
pp
pp
bpWa
2
1
111
Layer 2: Competition
The second layer is initialized with the output of the first layer.
The neurons compete with each other to determine a winner.
The neuron with the largest initial value will win the competition. winner-take-all
12 )0( aa
)(poslin)1( 222 tt aWa
1
10
otherwise.,
.if,12
S
jiwij
ij
jii tataposlinta )()()1( 222
Lateral Inhibition: excites itself and inhibits all the other neurons
Competitive Layer
A recurrent competitive layer:(assuming vectors have normalized length of L)
)(ncompeta
ST
S
T
T
L
L
L
cos
cos
cos
2
22
12
2
1
p
w
w
w
Wpn
ii
iiai
,0
,1
}...,,,max{ 21 Sinnnn
Competitive Learning
Instar rule: Train the weights in a competitive network, without
knowing the prototype vectors. For the competitive network, a is nonzero (i.e., 1) for
the winning neuron (i = i*). Kohonen rule:
)1()()()1()( qqqaqq iiii wpww
)()1()1(
)1()()1()(
qqqq
i
iii
pw
wpww
iiq
iiqqq
i
ii
),1(
),()1()1()(
w
pww
P14.2
P1= p2= p3= The initial value of the three weight vectors are
1w= 2w = 3w =
Calculate the resulting weights found after training the competitive layer with the Kuhonen rule and a learning rate a of 0.5, on the following series of inputs: p1 p2 p3 p1 p2 p3
0
1
2/1
2/1
1
0
0
1
5/1
5/2
5/2
5/1
Example: Hamming Network
Input vectors After p2 is presented Final weights& initial weights
Each weigh vector will point at a differentcluster of input vectors and become aprototype for a different cluster.
Once the network has learned to cluster the input vectors, it will classify new vectors accordingly.
The First Problem: The choice of learning rate forces a
trade-off between the speed of learningrate and the stability of the final weightvectors.
Initial training can be done with a largelearning rate for fast learning. Then thelearning rate can be decreased astraining progressed, to achieve stableprototype vectors.
Prob. with Compet. Layer
The Second Problem: A more serious stability problem occurs when
clusters are close together.
Input vector: blue star; Order: (a) (b) (c) Two input vectors in (a) are presented several times.
The final weight vectors are as (b), and so on. Resulting in unstabe learning.
Prob. with Compet. Layer
Prob. with Compet. Layer
The Third Problem: A neuron’s initial weight vector is
located so far from any input vectorthat it never wins the competition,and therefore never learns.
Resulting in a “dead” neuron,which does nothing useful.
Prob. with Compet. Layer
The Forth Problem: When the number of clusters is not known in
advance, the competitive layer may not acceptable for applications.
The Fifth Problem: Competitive layers cannot form classes with
nonconvex regions or classes that are the union of unconnected regions.
Compet. Layers in Biology
On-center/off-surround connectionOn-center/off-surround connection The weights in layer 2 of the Hamming network:
In terms of the distances:
Each neuron reinforces itself (center),while inhibiting all other neurons (surround).
ji
jiwij if,
if,1
0if,
0if,1
ij
ij
ij d
dw
Mexican-Hat Function
In biology, a neuron reinforces not only itself, but also those neurons close to it.
Typically, the transition from reinforcement to inhibition occurs smoothly as the distance between neurons increases.
Neighborhood
The neighborhood Ni(d) contains the indices for all the neurons that lie within a radius d of the neuron i.
},{)( ddjdN iji
首先由 Kohonen 提出，屬於前饋式、非監督式神經網路 以特徵映射的方式，將任意維度的輸入向量，映射至較低維（度）
的特徵映射圖上
自組特徵映射網路架構與其演算法自組特徵映射網路（自組特徵映射網路（ Self-Organizing Map, SOMSelf-Organizing Map, SOM ） ）
二維矩陣的二維矩陣的 SOM SOM 架構圖 架構圖
而最後輸出層的神經元會依據輸入向量的「特徵」以有意義的「拓樸結構」（ topological structure）展現在輸出空間中，
由於所產生的拓樸結構圖可以反應所有輸入值間的分布關係，因此將此網路稱作為自組特徵映射網路，而該映射圖也可稱為拓樸圖（ topology）。
SOMSOM 網路神經元間的拓樸座標 網路神經元間的拓樸座標
• 依據目前的輸入向量在神經元間彼此相互競爭，優勝的神經元可獲依據目前的輸入向量在神經元間彼此相互競爭，優勝的神經元可獲得調整連結權重向量的機會；得調整連結權重向量的機會；
Self-Organizing Feature Map
Determine the winning neuron i* using the same procedure as the competitive layer
Update the weight vectors using the Kohonen rule:
)(),()1()1()( dNiqqqiii pww
2D topology
initial weight vectors
Uses
Example: Data sets for poverty levels in different countries. Data sets have many different statistics for each country. SOM does not show poverty levels, rather it shows how similar
the poverty sets for different countries are to each other. (Similar color = similar data sets).
→
Feature Map Training
250 iterations per diagram
Other Training Examples
Improving Feature Maps
To speed up the self-organizing process and to make it more reliable
Gradually reduce the size of the neighborhoods during training until it only covers the winning neuron.
Gradually decrease the learning rate asymptotically toward 0 during training.
The winning neuron uses a larger learning rate than the neighboring neurons.
Learning Vector Quantization
LVQ network is a hybrid network. It uses both unsupervised and supervised learning to form classifications.
LVQ: Competitive Layer
Each neuron in the 1st layer isassigned to a class, with severalneurons often assigned to thesame class.
Each class is then assigned to oneneuron in the 2nd layer.
)( 11
1
12
11
111
1
ncompeta
pw
pw
pw
npw
S
iin
Negative distance
Subclass
In the competitive network, the neuron with the nonzero output indicates which class the input vector belongs to.
For the LVQ network, the winning neuron in the first layer indicates a subclass, rather than a class.
There may be several different neurons (subclasses) that make up each class.
LVQ: Linear Layer
The 2nd layer of the LVQ network is used to combine subclasses into a single class.
The columns of W2 represent subclasses,and the rows represent classes.
W2 has a single 1 in each column,with the other elements set to zero. subclass i is a part of class k12
kiw
01000
10010
001012W
Subclasses 1 & 3 belong to class 1Subclasses 2 & 5 belong to class 2Subclass 4 belongs to class 3
Complex/Convex Boundary
A standard competitive layer has the limitation that it can only create decision regions that are convex.
The process of combining subclasses to form a class allows the LVQ network to create complex class boundaries.
LVQ Learning
The learning in the LVQ network combines competitive learning with supervision.
Assignment of W2:
If hidden neuron i is to be assigned to class k, then set If p is classified correctly, move the weights
of the winning hidden neuron toward p.
If p is classified incorrectly, move the weights of the winning hidden neuron away p.
1wi
)1()()1()( 111 qqqqiii
wpww 1wi
)1()()1()( 111 qqqqiii
wpww
12 kkta
kkta2
12 kiw
Example: LVQ
Classification problem:
1
1
1
1
1
1
1
1
T0121 tt
T1043 tt
840.0
543.0
249.0
969.0
094.0
997.0
954.0
456.0
1100
00112W
Training Process
Present p3 to the network:
0
1
0
0
03.2
09.1
11.2
40.2
)(
14
13
12
11
11 compet
pw
pw
pw
pw
competncompeta
3122
1
0
0
1
0
0
1100
0011taWa
453.0
998.0)0(5.0)0()1( 1
331
31
3 wpww
After 1st & Many Iterations
Improving LVQ networks
First, as with competitive layers, occasionally a hidden neuron may be a dead neuron.
Secondly, depending on how initial weight vectors are arrange, a neuron’s weight vector may have to travel through a region of a class that it doesn’t represent, to get to a region that it does represent.
The second problem can be solved by applying the modification to the Kohonen rule (LVQ2).
LVQ2
When the network correctly classifies an input vector, the weights of only one neuron are moved toward the input vector.
If the input vector is incorrectly classified, the weights of two neurons are updated, one weight vector is move away from the input vector, and the other one is moved toward the input vector.