neural networks for optimization
DESCRIPTION
Neural Networks for Optimization. Bill Wolfe California State University Channel Islands. Neural Models. Simple processing units Lots of them Highly interconnected Exchange excitatory and inhibitory signals Variety of connection architectures/strengths - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/1.jpg)
Neural Networks for
OptimizationBill Wolfe
California State University Channel Islands
![Page 2: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/2.jpg)
![Page 3: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/3.jpg)
Neural Models
• Simple processing units• Lots of them• Highly interconnected• Exchange excitatory and inhibitory signals• Variety of connection architectures/strengths• “Learning”: changes in connection strengths• “Knowledge”: connection architecture• No central processor: distributed processing
![Page 4: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/4.jpg)
Simple Neural Model
• ai Activation
• ei External input
• wij Connection Strength
Assume: wij = wji (“symmetric” network)
W = (wij) is a symmetric matrix
ai ajwij
ei ej
![Page 5: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/5.jpg)
Net Input
eaWnet
i
j
jiji eawnet ai
aj
wij
Vector Format:
![Page 6: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/6.jpg)
Dynamics
• Basic idea:
ai
neti > 0
ai
neti < 0
ii
ii
anet
anet
0
0
netdt
adnet
dt
dai
i
![Page 7: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/7.jpg)
Energy
aeaWaE TT 21
![Page 8: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/8.jpg)
net
netnet
ewew
aEaE
E
n
j
nnj
j
j
n
,...,
,...,
/,....,/
1
11
1
netE
![Page 9: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/9.jpg)
Lower Energy
• da/dt = net = -grad(E) seeks lower energy
net
Energy
a
![Page 10: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/10.jpg)
Problem: Divergence
Energy
net a
![Page 11: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/11.jpg)
A Fix: Saturation
))(1( iiii
aanetdt
da
corner-seeking
lower energy
10 ia
![Page 12: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/12.jpg)
Keeps the activation vector inside the hypercube boundaries
a
Energy
0 1
))(1( iiii
aanetdt
da
corner-seeking
lower energy
Encourages convergence to corners
![Page 13: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/13.jpg)
Summary: The Neural Model
))(1( iiii
aanetdt
da
i
j
jiji eawnet
ai Activation ei External Inputwij Connection StrengthW (wij = wji) Symmetric
10 ia
![Page 14: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/14.jpg)
Example: Inhibitory Networks
• Completely inhibitory– wij = -1 for all i,j– k-winner
• Inhibitory Grid– neighborhood inhibition
![Page 15: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/15.jpg)
Traveling Salesman Problem
• Classic combinatorial optimization problem
• Find the shortest “tour” through n cities
• n!/2n distinct tours
D
D
AE
B
C
AE
B
C
ABCED
ABECD
![Page 16: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/16.jpg)
TSP solution for 15,000 cities in Germany
![Page 17: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/17.jpg)
TSP
50 City Example
![Page 18: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/18.jpg)
Random
![Page 19: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/19.jpg)
Nearest-City
![Page 20: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/20.jpg)
2-OPT
![Page 21: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/21.jpg)
http://www.jstor.org/view/0030364x/ap010105/01a00060/0
An Effective Heuristic for the Traveling Salesman Problem
S. Lin and B. W. Kernighan
Operations Research, 1973
![Page 22: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/22.jpg)
Centroid
![Page 23: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/23.jpg)
Monotonic
![Page 24: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/24.jpg)
Neural Network Approach
D
C
B
A1 2 3 4
time stops
cities neuron
![Page 25: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/25.jpg)
Tours – Permutation Matrices
D
C
B
A
tour: CDBA
permutation matrices correspond to the “feasible” states.
![Page 26: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/26.jpg)
Not Allowed
D
C
B
A
![Page 27: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/27.jpg)
Only one city per time stopOnly one time stop per city
Inhibitory rows and columns
inhibitory
![Page 28: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/28.jpg)
Distance Connections:
Inhibit the neighboring cities in proportion to their distances.
D
C
B
A-dAC
-dBC
-dDC
D
A
B
C
![Page 29: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/29.jpg)
D
C
B
A-dAC
-dBC
-dDC
putting it all together:
![Page 30: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/30.jpg)
Research Questions
• Which architecture is best?• Does the network produce:
– feasible solutions?– high quality solutions?– optimal solutions?
• How do the initial activations affect network performance?
• Is the network similar to “nearest city” or any other traditional heuristic?
• How does the particular city configuration affect network performance?
• Is there a better way to understand the nonlinear dynamics?
![Page 31: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/31.jpg)
A
B
C
D
E
F
G
1 2 3 4 5 6 7
typical state of the network before convergence
![Page 32: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/32.jpg)
“Fuzzy Readout”
A
B
C
D
E
F
G
1 2 3 4 5 6 7
à GAECBFD
A
B
C
D
E
F
G
![Page 33: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/33.jpg)
Neural ActivationsFuzzy Tour
Initial Phase
![Page 34: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/34.jpg)
![Page 35: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/35.jpg)
Neural ActivationsFuzzy Tour
Monotonic Phase
![Page 36: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/36.jpg)
Neural ActivationsFuzzy Tour
Nearest-City Phase
![Page 37: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/37.jpg)
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9
8
7
6
to
ur
len
gt
h
10009008007006005004003002001000iteration
Fuzzy Tour Lengths
centroidphase
monotonicphase
nearest-cityphase
monotonic (19.04)
centroid (9.76)nc-worst (9.13)
nc-best (7.66)2opt (6.94)
Fuzzy Tour Lengthstour length
iteration
![Page 38: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/38.jpg)
12
11
10
9
8
7
6
5
4
3
2
tour length
70656055504540353025201510# cities
average of 50 runs per problem size
centroid
nc-w
nc-bneur
2-opt
Average Results for n=10 to n=70 cities
(50 random runs per n)
# cities
![Page 39: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/39.jpg)
DEMO 2
Applet by Darrell Longhttp://hawk.cs.csuci.edu/william.wolfe/TSP001/TSP1.html
![Page 40: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/40.jpg)
Conclusions
• Neurons stimulate intriguing computational models.
• The models are complex, nonlinear, and difficult to analyze.
• The interaction of many simple processing units is difficult to visualize.
• The Neural Model for the TSP mimics some of the properties of the nearest-city heuristic.
• Much work to be done to understand these models.
![Page 41: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/41.jpg)
EXTRA SLIDES
![Page 42: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/42.jpg)
E = -1/2 { ∑i ∑x ∑j ∑y aix ajy wixjy }
= -1/2 {
∑i ∑x ∑y (- d(x,y)) aix ( ai+1 y + ai-1 y) + ∑i ∑x ∑j (-1/n) aix ajx + ∑i ∑x ∑y (-1/n) aix aiy +
∑i ∑x ∑j ∑y (1/n2) aix ajy
}
![Page 43: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/43.jpg)
wix jy =
1/n2 - 1/n y = x, j ≠ i; (row) 1/n2 - 1/n y ≠ x, j = i; (column)
1/n2 - 2/n y = x, j = i; (self)1/n2 - d(x, y) y ≠ x, j = i +1, or j = i - 1. (distance )
1/n2 j ≠ i-1, i, i+1, and y ≠ x; (global )
![Page 44: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/44.jpg)
Brain
• Approximately 1010 neurons• Neurons are relatively simple• Approximately 104 fan out• No central processor• Neurons communicate via excitatory and
inhibitory signals• Learning is associated with modifications of
connection strengths between neurons
![Page 45: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/45.jpg)
Fuzzy Tour Lengths
iteration
tour length
![Page 46: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/46.jpg)
Average Results for n=10 to n=70 cities
(50 random runs per n)
# cities
tour length
![Page 47: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/47.jpg)
01
10W
1,1
1
1,1
1
v
v
a1
a2
1
1
0111
1011
1101
1110
W
![Page 48: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/48.jpg)
a1
a2
1
1
with external input e = 1/2
![Page 49: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/49.jpg)
Perfect K-winner Performance: e = k-1/2
e
e
e
e
e
e
e
e
![Page 50: Neural Networks for Optimization](https://reader036.vdocument.in/reader036/viewer/2022062723/56813fce550346895daaab1d/html5/thumbnails/50.jpg)
1
0
initial activations
final activations
1
0
initial activations
final activations
e=½(k=1)
e=1 + ½(k=2)