self-organizing feature maps week 7cosc.brocku.ca/~efoxwell/4p80/week7.pdf ·...
TRANSCRIPT
COSC 4P80
Self-Organizing Feature Maps
Week 7
Brock University(Material attributed to Rojas, 1996)
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 1 / 17
Recall clustering...
A category of unsupervised learning techniques that, after training onpatterns, can then arrange additional patterns.
Patterns that are more similar — according to some metric — aremore likely to be grouped together than fundamentally dissimilarpatterns
Effectively, like with like
We would prefer to see topologies preserved: clusters with mostlysimilar characteristics could also be close to each other
◮ Though we haven’t really addressed how to accomplish this just yet...
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 2 / 17
Consider a problem
We’ve already discussed the idea of taking a large sampling ofrandomly-selected colours, and grouping them together into whateverarrangement the algorithm might choose
e.g. for K-Means Clustering
Maybe we could do an incredibly brief pseudo-example on the board?
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 3 / 17
Colours are prettyBut how do we handle similarities?
Let’s take it one step further, and consider topologies
That is, not only assigning the patterns to nodes/vectors, but alsoarranging the vectors to nudge adjacent nodes towards preservingrelative positioning, etc.
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 4 / 17
ExampleColours
Consider the following colours, corresponding to different neurons/vectors
That is, these are the network; not the input patterns
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 5 / 17
ExampleColours (cont)
Through some form of wizardry, we’d like to see something like thishappen:
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 6 / 17
ExampleColours (additional)
Note: some settings were tweaked there to show the progression on aprojector. This would actually probably be a better result:
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 7 / 17
Topologies
I keep saying “Like with Like”, but what does that mean?
It means that we’re actually preserving the mathematical similaritieswhen mapping a set of data into another field
◮ Irrespective of whether we’re maintaining or reducing the number ofdimensions
For example:
Credit: http://commons.wikimedia.org/wiki/User:Mcld
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 8 / 17
Our Approach
Our algorithm will:
Have a ‘network’ that’s mostly just a collection of vectors mappingfrom inputs
Use its vectors as exemplars reflecting their correspondingclusters/nodes
For a given pattern, choose the vector that most closely resemblesthat pattern
Update its exemplars to more better reflect the cluster’s members
Wait... isn’t that just K-Means clustering?!?
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 9 / 17
Our ApproachThe difference
No! Because now we’re going to introduce one more feature: aneighbourhood!
Remember that we’d like to incorporate some notion of topologies
We achieve this by, during the training phase, acknowledging all ofthe other nodes that might have been chosen
◮ e.g. adjacent nodes, or even just nodes that were reasonably close
In addition to training the selected node, we’ll also be training thosenodes within the neighbourhood!
◮ Of course, it’s worth pointing out that the selected node will normallybe trained more than those that were simply close
The net result is exemplars that reflect their members very well, andadjacent nodes that also bear some resemblance
◮ i.e. similar in input space → similar in clustering
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 10 / 17
Disclaimer
This is where we could discuss the biological reasoning behind thisapproach, and point out the similarities with how the human brain tendsto associate information, etc.
But personally I think that’s immaterial
The real reason we’ll be using this is because it works
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 11 / 17
Self-Organizing Feature MapsKohonen Networks
Teuvo Kohonen came up with this nifty idea. It’s really just an extensionof what we’ve already seen.
We can have our nodes arranged in a line, 2D grid, or any otherdimensionality, but 2D’s pretty common
We’ll need to define a way of deciding the size of a neighbourhood◮ Since we’ll want it to finetune eventually, the radius of that
neighbourhood will shrink over time, until it includes only the selectednode
For the same reason, we’ll apply a learning rate, but also reduce thatover time
Of course, since our formulae will have a built-in mechanism forremoving training (after some number of epochs), it’s mathematicallycertain to eventually converge
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 12 / 17
SOMs — The MathWeight Update Rule:
~w t+1i
= ~w t
i + φ(t + 1)η(t + 1)(~v − ~w t
i )
Neighbourhood Multiplier:
φ(t) = e−
dist2
2σ2(t) where dist is the distance from the node to the BMU
Neighbourhood Size:σ(t) = σ0e
−
t
λ
Learning Rate:η(t) = η0e
−
t
λ
Distance between nodes:
dist =
√
√
√
√
i=n∑
i=0
(Vi −Wi )2
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 13 / 17
Applications
Of course, SOMs can be used for anything requiring clustering. Though,because they’re so easy to implement and effective, they’ve been appliedto a wide variety of problems.
They can often be used as a sort of ‘pre-learning’ tool◮ Funnel the data through a Kohonen network, and then apply a different
learning algorithm onto that data!
Reduction of dimensionality◮ Related to the previous point, SOMs can be a good way to transform
data⋆ Sometimes important features may be lost, but sometimes it simplifies
the problem
You might want to take a glance here:http://www.ai-junkie.com/ann/som/som5.html
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 14 / 17
See Also...
You might also be interested in researching neural gas, a related topic.
Heck, it might even make a good seminar topic!◮ Hint
⋆ (hint)
◮ Then again, Learning Vector Quantization (LVQ) might as well...⋆ (hint hint)
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 15 / 17
Additional Reading
Of course, the (Rojas) book
AI Junkie: http://www.ai-junkie.com/ann/som/som1.html
https://www.cs.hmc.edu/~kpang/nn/som.html
https://www.hindawi.com/journals/cin/2017/4263064/
http://www.mperfect.net/aisompic/
◮ Note: gets a bit... creepy towards the latter half
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 16 / 17
Questions?Comments?
Funny anecdotes?
Brock University(Material attributed to Rojas, 1996) (Week 7)Self-Organizing Feature Maps 17 / 17