statistical clustering
TRANSCRIPT
UNCL
ASSI
FIED
Statistical Clustering: k-means, Gaussian Mixtures, Variational Inference
22-FEB-2012
UNCL
ASSI
FIED
What is Clustering?
22FE
B12
2 Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
Design Considerations• Features• Dimension• Model: Distance / Cost• Bias / Variance
UNCL
ASSI
FIED
Why do we care?
22FE
B12
3 Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
UNCL
ASSI
FIED
Scope of Talk – Main Take Away Point
22FE
B12
4 Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
It’s all About the Posterior
K-meansHow does it workMath behind itIssues
GMMHow does it workMath behind itIssues
VariationalJust the facts
Variational InferenceGMM, EM, (Graph Cuts, Spectral Clustering)K-means, vector quantization
UNCL
ASSI
FIED
Scope of Talk
22FE
B12
5 Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
Main Take Away PointIt’s all Just Posterior EstimationVariational / MCNCGMMK-means / vector quantization
K-meansHow does it workMath behind itIssues
GMMHow does it workMath behind itIssues
VariationalJust the factsPlease interrupt
and ask
questions
UNCL
ASSI
FIED
K-means – How it works
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
6
Goal: represent a data set in terms of K clusters each of which is summarized by a prototype Iterative Two step process:E-step: assign each data point to nearest prototypeM-step: update prototype to be the cluster meansSimple version: Euclidean distance, requires whitening
Design Considerations• Features• Dimension• Model: Distance / Cost• Bias / Variance
UNCL
ASSI
FIED
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
7
UNCL
ASSI
FIED
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
8
UNCL
ASSI
FIED
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
9
UNCL
ASSI
FIED
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
10
UNCL
ASSI
FIED
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
11
UNCL
ASSI
FIED
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
12
UNCL
ASSI
FIED
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
13
UNCL
ASSI
FIED
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
14
UNCL
ASSI
FIED
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
15
Converged
UNCL
ASSI
FIED
k-means - Math Responsibilities – assign data to cluster
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
16
Cost Function
example
UNCL
ASSI
FIED
Minimizing the Cost Function
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
17
UNCL
ASSI
FIED
What can go wrong?
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
18
UNCL
ASSI
FIED
What can go wrong? A great deal. How do we choose K? (gap statistic / prediction strength) How do we initialize? (k++ seems to be the best) Local minimums – run hundreds of time with different
initializations Are we overfitting? Probably. But hey – it simple to understand and does not cost too
many cycles
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
19
UNCL
ASSI
FIED
Quick word on distances (k-medioids)
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
20
MahalanobisNot dependent on scale of measurementTuning parameter
Manhattan / City BlockDampens outliers
EuclideanNeed to whitenOutliers are an issue
UNCL
ASSI
FIED
Exclusive Clustering: k-means, weighted k-means Overlapping Clustering: fuzzy c-means, Nonlinear Clustering: kernel k-means (spectral clustering,
normalized cuts)
Hierarchical Clustering: Hierarchical
Quicker word on flavors
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
21
UNCL
ASSI
FIED
Probabilistic Clustering Represent the probability distribution of the data as a
mixture model Captures uncertainty in cluster assignments Gives model for data distribution Bayesian mixture – we can figure out K easier
Consider a mixture of Gaussians
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
22
UNCL
ASSI
FIED
Multivariate Gaussian Distribution Review
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
23
UNCL
ASSI
FIED
Likelihood Function
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
24
Maximum LikelihoodWhat is the best fit to my dataApproximation of Posterior!
UNCL
ASSI
FIED
Maximum Likelihood Solution for One Gaussian Sample mean
Sample Covariance
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
25
UNCL
ASSI
FIED
Gaussian Mixtures Linear super-position of Gaussians
Normalization and positivity require
Can interpret mixing coefficients as prior probabilities
[Aside]We can sample from this. Given mixing coeff, mean, variance – get a sample from p(x) – our dataset.22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
26
UNCL
ASSI
FIED
Fitting the Gaussian Mixture We wish to invert this sampling process – given the data,
find the corresponding parameters (like we did for the single Gaussian case) Mixing coefficients Means Covariances
If we knew which data point “belonged” or was the responsibility of which Gaussian, then we could use our single Gaussian ML solution
Problem: We don’t have labels, this complicates things. Solution: Create a latent or hidden variable (z) that tells
us which data point goes with which Gaussian22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
27
UNCL
ASSI
FIED
Posterior of latent variable Or more concretely the probability that the data point
was generated by the Gaussian with no prior knowledge of .
Or more concretely the probability that the data point was generated by the Gaussian after observing
Also called responsiblities
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
28
UNCL
ASSI
FIED
Maximum Likelihood for GMM The log likelihood takes this form
Notice that the sum inside the log, no closed form solution.
Solve by expectation-maximization (EM) algorithm Derivative w.r.t
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
29
UNCL
ASSI
FIED
EM – notice each one of these is dependent on responsiblities Do the Same for Covariance
Use Lagrange Multiplier for mixing coefficients
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
30
UNCL
ASSI
FIED
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
31
UNCL
ASSI
FIED
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
32
UNCL
ASSI
FIED
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
33
UNCL
ASSI
FIED
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
34
UNCL
ASSI
FIED
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
35
UNCL
ASSI
FIED
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
36
UNCL
ASSI
FIED
Relation to k-means
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
37
UNCL
ASSI
FIED
Fast food example
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
38
http://nutrition.mcdonalds.com/nutritionexchange/nutritionfacts.pdf
UNCL
ASSI
FIED
Dessert Cluster
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
39
Caramel MochaFrappe CaramelIced Hazelnut LatteIced CoffeeStrawberry Triple Thick Shake
Snack Size McFlurryHot Caramel SundaeBaked Hot Apple PieCinnamon MeltsKiddie ConeStrawberry Sundae
UNCL
ASSI
FIED
Burger – like cluster
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
40
HamburgerCheeseburgerFilet-O-FishQuarter Pounder with CheesePremium Grilled Chicken Club Sandwich
Ranch Snack WrapPremium Asian Salad with Crispy ChickenButter Garlic CroutonsSausage McMuffinSausage McGriddles
UNCL
ASSI
FIED
Salad Cluster
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
41
Premium Southwest Salad with Grilled ChickenPremium Caesar Salad with Grilled ChickenSide SaladPremium Asian Salad without ChickenPremium Bacon Ranch Salad without Chicken
UNCL
ASSI
FIED
Sauces Cluster 2 /6
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
42
Hot Mustard SauceSpicy Buffalo SauceNewman’s Own Low Fat Balsamic Vinaigrette
Ketchup PacketBarbeque SauceChipotle Barbeque Sauce
UNCL
ASSI
FIED
Creamy Sauces
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
43
Creamy Ranch SauceNewman’s Own Creamy Caesar DressingCoffee CreamIced Coffee with Sugar Free Vanilla Syrup
UNCL
ASSI
FIED
Oatmeal and Apples on their own
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
44
UNCL
ASSI
FIED
Breakfast artery clogging cluster
22FE
B12
Notice: Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document.
45
Sausage McMuffin with EggSausage BurritoEgg McMuffinBacon, Egg & Chees BiscuitMcSkillet Burrito with SausageBig Breakfast with Hotcakes