lirong xia kernel (continued), clustering friday, april 18, 2014
TRANSCRIPT
Lirong Xia
Kernel (continued), Clustering
Friday, April 18, 2014
• Project 4 due tonight
• Project 5 (last project!) out later
– Q1: Naïve Bayes with auto tuning (for smoothing)
– Q2: freebie
– Q3: Perceptron
– Q4: another freebie
– Q5: MIRA with auto tuning (for C)
– Q6: feature design for Naïve Bayes
2
Projects
Last time
3
• Fixing the Perceptron: MIRA
• Support Vector Machines
• k-nearest neighbor (KNN)
MIRA
4
• MIRA*: choose an update size that fixes the current mistake
• …but, minimizes the change to w
• Solution
*Margin Infused Relaxed Algorithm
MIRA with Maximum Step Size
5
• In practice, it’s also bad to make updates that are too large– Solution: cap the maximum possible
value of τ with some constant C
– tune C on the held-out data
Support Vector Machines
6
• Maximizing the margin: good according to intuition, theory, practice
SVM
Parametric / Non-parametric
7
• Parametric models:– Fixed set of parameters– More data means better settings
• Non-parametric models:– Complexity of the classifier increases with
data– Better in the limit, often worse in the non-limit
• (K)NN is non-parametric
Basic Similarity
8
• Many similarities based on feature dot products:
• If features are just the pixels:
• Note: not all similarities are of this form
Perceptron Weights
9
• What is the final value of a weight wy of a perceptron?– Can it be any real vector?– No! it’s build by adding up inputs
• Can reconstruct weight vectors (the primal representation) from update counts (the dual representation)
1 5
,
0 ...y
y i y ii
w f x f xw a f x
primal
dual
Dual Perceptron
10
• Start with zero counts (alpha)• Pick up training instances one by one
• Try to classify xn,
• If correct, no change!• If wrong: lower count of wrong class (for this
instance), raise score of right class (for this instance)
• The kernel trick
• Neuron network
• Clustering (k-means)
11
Outline
Kernelized Perceptron
12
• If we had a black box (kernel) which told us the dot product of two examples x and y:– Could work entirely with the dual representation– No need to ever take dot products (“kernel trick”)
• Like nearest neighbor – work with black-box similarities
• Downside: slow if many examples get nonzero alpha
Kernelized Perceptron Structure
13
,
= score ,
i c i
y x
a
Kernels: Who Cares?
14
• So far: a very strange way of doing a very simple calculation
• “Kernel trick”: we can substitute many similarity function in place of the dot product
• Lets us learn new kinds of hypothesis
Non-Linear Separators
15
• Data that is linearly separable (with some noise) works out great:
• But what are we going to do if the dataset is just too hard?
• How about… mapping data to a higher-dimensional space:
This and next few slides adapted from Ray Mooney, UT
Non-Linear Separators
16
• General idea: the original feature space can always be mapped to some higher-dimensional feature space where the training set is separable:
Some Kernels
17
• Kernels implicitly map original vectors to higher dimensional spaces, take the dot product there, and hand the result back
• Linear kernel:
• Quadratic kernel:
• RBF: infinite dimensional representation
• Mercer’s theorem: if the matrix A with Ai,j=K(xi,xj) is positive definite, then K can be represented as the inner product of a higher dimensional space
Why Kernels?
18
• Can’t you just add these features on your own?– Yes, in principle, just compute them– No need to modify any algorithms– But, number of features can get large (or infinite)
• Kernels let us compute with these features implicitly– Example: implicit dot product in quadratic kernel takes
much less space and time per dot product– Of course, there’s the cost for using the pure dual
algorithms: you need to compute the similarity to every training datum
• The kernel trick
• Neuron network
• Clustering (k-means)
19
Outline
• Neuron = Perceptron with a non-linear
activation function
ai=g(ΣjWj,iaj)
20
Neuron
• (a) step function or threshold function
– perceptron
• (b) sigmoid function 1/(1+e-x)
– logistic regression
• Does the range of g(ini) matter? 21
Activation function
• AND, OR, and NOT can be implemented
• XOR cannot be implemented
22
Implementation by perceptron
• Feed-forward networks
– single-layer (perceptrons)
– multi-layer (perceptrons)
• can represent any boolean function
– Learning
• Minimize squared error of prediction
• Gradient descent: backpropagation algorithm
• Recurrent networks
– Hopfield networks have symmetric weights
Wi,j=Wj,i
– Boltzmann machines use stochastic activation
functions
23
Neural networks
• a5=g5 (W3,5×a3+W4,5×a4)
=g5 (W3,5×g3 (W1,3×a1+W2,3×a2)+W4,5×g4 (W1,4×a1+W2,4×a2))
24
Feed-forward example
• The kernel trick
• Neuron network
• Clustering (k-means)
25
Outline
Recap: Classification
26
• Classification systems:– Supervised learning– Make a prediction given
evidence– We’ve seen several
methods for this– Useful when you have
labeled data
Clustering
27
• Clustering systems:– Unsupervised learning– Detect patterns in unlabeled
data• E.g. group emails or search
results
• E.g. find categories of customers
• E.g. detect anomalous program executions
– Useful when don’t know what you’re looking for
– Requires data, but no labels– Often get gibberish
Clustering
28
• Basic idea: group together similar instances• Example: 2D point patterns
• What could “similar” mean?– One option: small (squared) Euclidean distance
2dist ,
T
i iix y x y x y x y
K-Means
29
• An iterative clustering algorithm– Pick K random points
as cluster centers (means)
– Alternate:• Assign data instances
to closest mean
• Assign each mean to the average of its assigned points
– Stop when no points’ assignments change
K-Means Example
30
Example: K-Means
31
• [web demo]– http://www.cs.washington.edu/research/imagedatabase/
demo/kmcluster/
K-Means as Optimization
32
• Consider the total distance to the means:
• Each iteration reduces phi• Two states each iteration:
– Update assignments: fix means c, change assignments a
– Update means: fix assignments a, change means c
dis, = ,, tii i ik i ax c x ca
points assignments means
Phase I: Update Assignments
33
• For each point, re-assign to closest mean:
• Can only decrease total distance phi!
Phase II: Update Means
34
• Move each mean to the average of its assigned points
• Also can only decrease total distance…(Why?)
• Fun fact: the point y with minimum squared Euclidean distance to a set of points {x} is their mean
Initialization
35
• K-means is non-deterministic– Requires initial means– It does matter what you pick!
– What can go wrong?
– Various schemes for preventing this kind of thing: variance-based split / merge, initialization heuristics
K-Means Getting Stuck
36
• A local optimum:
K-Means Questions
37
• Will K-means converge?– To a global optimum?
• Will it always find the true patterns in the data?– If the patterns are very very clear?
• Will it find something interesting?
• Do people ever use it?
• How many clusters to pick?
Agglomerative Clustering
38
• Agglomerative clustering:– First merge very similar instances– Incrementally build larger clusters out
of smaller clusters
• Algorithm:– Maintain a set of clusters– Initially, each instance in its own
cluster– Repeat:
• Pick the two closest clusters• Merge them into a new cluster• Stop when there is only one cluster left
• Produces not one clustering, but a family of clusterings represented by a dendrogram
Agglomerative Clustering
39
• How should we define “closest” for clusters with multiple elements?
• Many options– Closest pair (single-link clustering)– Farthest pair (complete-link
clustering)– Average of all pairs– Ward’s method (min variance, like k-
means)
• Different choices create different clustering behaviors
Clustering Application
40