1
Class #3: Clustering
ML4Bio 2012 January 27th, 2012
Quaid Morris
2 Module #: Title of Module
Overview • Objective functions • Parametric clustering (i.e. weʼre estimating
some parameters): – K-means – Gaussian mixture models
• Network-based (non-parametric clustering, no “parameters” estimated): – Hierarchical clustering – Affinity propagation – Graph (i.e., network) cuts – MCL 3
Advanced topics, next QM tutorial
Objective functions
4
An objective function, e.g., E(Θ), measures the fit of one or more parameters, indicated by Θ, to a set of observations.
By maximizing the objective function, we can do estimation by finding the settings of the parameters that have the best fit.
Likelihood and log likelihood are examples of common objective functions.
Notes about objective functions
5
Beware: Sometimes you are supposed to minimize objective functions (rather than maximizing them) but in those cases, the objective function is usually called a cost function or an error function.
Note: you can always turn a minimization problem into a maximization one by putting a minus sign in front of the cost function!
similarity = - distance
Examples of objective functions: log likelihood
• Estimating the bias of a coin, p, given that weʼve observed m heads and n tails.
Pr(m heads and n tails | bias of p) =
6
Examples of objective functions: log likelihood
• Estimating the bias of a coin, p, given that weʼve observed m heads and n tails.
Pr(m heads and n tails | bias of p) =
7
€
m + nm
pm(1− p)n
Examples of objective functions: log likelihood
• Estimating the bias of a coin, p, given that weʼve observed m heads and n tails.
Pr(m heads and n tails | bias of p) =
Use log likelihood minus a constant as objective function:
Maximum likelihood (ML) estimate:
8
€
m + nm
pm(1− p)n
€
E(p) =m log p + n log(1− p)
€
pML = argmax p E(p)
Examples of objective functions: sum of squared errors
• Estimating the mean of a distribution, m, given that weʼve observed samples x1, x2, …, xN.
Objective function:
Minimum sum of squared error (MSSE) estimate:
9
€
E(m) = − (i=1
N
∑ m − xi )2
€
mMSSE = argmaxm E(m)
A MSSE estimate is a ML estimate!
• The MSSE is the log likelihood (minus a constant) of the data under a Normal distribution with fixed variance:
Recall:
and if xi is normally distributed with mean m and variance of 1 (i.e. σ2 = 1), then:
10
€
€
P(x1,x2,…,xN |m) = P(xi |m)i=1
N
∏
€
P(xi |m) =12πe−(xi −m)
2 / 2
Sum of squared errors with vectors
• Recall from linear algebra that for vectors v, and w (where vj and wj are elements of these vectors) that their dot, or inner, product is:
• For measuring SSE between vector-valued parameters m and observations x1 , x2, etc. we use:
11
€
vTw = v jw jj∑
€
m − xii∑
2= (m − xi )
T (m − xi )i∑
Squared Euclidean distance
K-means • Given a “K”, the number of clusters and a set
of data vectors x1, x2, …, xN, find K clusters defined by their means (or sometimes, “centroids”): m1, m2, …, mK
Each data vector xi, is assigned to its closest mean. Let c(i) be the cluster that xi is assigned to, so:
12
€
c(i) = argmin j xi −mj2
K-means • Objective function:
sometimes E is called the distortion.
Recall:
13
€
c(i) = argmin j xi −mj2
€
E(m1,…mK ) = xi −mc(i )i∑
2
Lloydʼs algorithm for K-means • The K-means objective function is multimodal
and its not exactly clear how to minimize it. Thereʼs a number of algorithms for this (see, e.g., kmeans() help in R).
• However, one algorithm, Lloydʼs is so commonly used itʼs often called “the K-means algorithm”.
sometimes E is called the distortion. 14
Lloydʼs algorithm for K-means • Step 0: initialize the means
– (can do this by randomly sampling the means, or by randomly assigning data points to mean)
• Step 1: Compute the means based on the cluster assignments (“M-step”): – mj = mean of all xi such that c(i) = j.
• Step 2: Recompute the cluster assignments, c(i) based on the new means.
• Step 3: If the assignments donʼt change, then you are done, otherwise go back to Step 1.
15
Another K-means algorithm that almost always works better.
(sequential K-means) • Step 0: initialize the means
– (can do this by randomly sampling the means, or by randomly assigning data points to mean)
• Step 1: Compute the means based on the cluster assignments (“M-step”): – mj = mean of all xi such that c(i) = j.
• Step 2: Recompute the cluster assignment for a random point, xi
• Step 3: If you are not done, go back to Step 1. 16
Clustering example
x1
x2
Step 0
Clustering example
x1
x2
Step 1 “M-step”
Clustering example
x1
x2
Step 2 “E-step”
Clustering example
x1
x2
Step 1 “M-step”
Clustering example
x1
x2
Step 2 “E-step”
Clustering example
x1
x2
Step 1 “M-step”
Clustering example
x1
x2
Step 2 “E-step”
We’re done!
Gaussian mixture models • Weʼre not capturing any information about the
“spread” of the data when we do K-means – this can be especially important when choosing # of means.
• Rarely can you eyeball the data like we just did to choose # of means. So, weʼd like to know if we have one cluster with a broad spread in one dimension and narrow in another or multiple clusters.
• This narrow/broad spread can happen if your dimensions are measured in different units. 24
Gaussian mixture model
x1
x2
What we want
Σ2
Σ1
Gaussian mixture model • Objective function:
where
26
€
E(m1,…mK,Σ1,…,ΣK ) =
P(xi m1,…mK,Σ1,…,ΣK ) |i∏
€
P(xi m1,…mK,Σ1,…,ΣK ) =
π jN(xi m j ,j∑ Σ j ) Multivariate normal
density
Whatʼs Σ??? • Σ is the covariance matrix, it specifies the
shape of the distribution • If Σ = σ2 I (the identity matrix), then the
distribution is circular, with variance σ2 in every direction.
• If Σ is a diagonal matrix, then the distribution is “axis-aligned” but may be elliptical.
• If Σ is neither, then the ellipse is slanted
27
Gaussian mixture model
x1
x2
Σ1 is the identity matrix
Gaussian mixture model
x1
x2
Σ1 is diagonal
Gaussian mixture model
x1
x2
Σ1
Σ1 is not diagonal
Hierarchical agglomerative clustering
Often difficult to determine correct number of clusters ahead of time and want to group observations at different levels of resolution
dendrogram
clustergram Eisen et al, PNAS 1998
Hierarchical agglomerative clustering
• Start with set of clusters : – C = {{x1}, {x2}, …, {xN}} each containing exactly
one of the observations, also assume a distance metric dij is defined for each xi and xj
• While not done: – Find most similar (i.e. least distant) pair of clusters
in C, say Ca & Cb – Merge Ca and Cb to make a new cluster Cnew,
remove Ca and Cb from C and add Cnew – done if C contains only one cluster
Hierarchical agglomerative clustering
Algorithms vary on how they calculate the distance of clusters d(Ca, Cb). In all cases, if both clusters contain only one item, say Ca={xi} and Cb={xj} then d(Ca, Cb) = dij
33
Hierarchical agglomerative clustering
If clusters have >1 item, then have to choose linkage criterion:
Average linkage (UPGMA): d(Ca, Cb) = mean of distances between items in clusters
Single linkage: d(Ca, Cb) = minimum distance between items in clusters
Complete linkage: d(Ca, Cb) = maximum distance between items in clusters
35
Drawing the dendrogram
Ca Cb
d(Ca,Cb)
This node represents Cnew
Rotation of the subtrees in the dendrogram is arbitrary.
dendrogram
clustergram Eisen et al, PNAS 1998
Advanced topic: ordering the items
Advanced topic: ordering the items
• When displaying the clustergram, the ordering needs to be consistent with the dendrogram (and the clustering) but there are many consistent orderings, as you can arbitrarily rotate the trees.
• Can use the “TreeArrange” (Ziv Bar-Joseph et al, 2001) algorithm to find the ordering of the items that minimizes the distance between adjacent items while being consistent with the dendrogram (and clustering).
Affinity propagation(Frey and Dueck, Science 2007)
Exemplar-based clustering method, i.e. the cluster centre is on one of the datapoints. Also, “automatically” chooses the number of cluster centres to use. Requires the similarities, sij, for each pair of data items xi and xj.
Affinity propagation • Objective function (for similarities si,j):
Ε(c) = Σi si,c(i) where c(i) is the index of the centre that data item xi is
assigned to. • The self-similarities sii can determine # of
centres, e.g., if sii is less than sij for all j (not equal to i), then there will be only one centre. If sii is greater than sij, for all j, all points will be there own centre.
Propagation algorithm for assessing c(i)
• Updates two sets of quantities: – rik , the responsibility of k for i – aik, the availability of k to serve as iʼs centre
rik = sik – maxkʼ | kʼ is not k (aikʼ + sikʼ) aik = min{0, rkk+ Σiʼ | iʼ is not i or k max(0,riʼk)} akk = Σiʼ | iʼ is not k max(0,riʼk)
• aik all initially start at 0 • c(i) = argmax rik + aik, once converged