introduction to clustering - uci mathematics€¦ · ‣ start with assigning all data points to...
TRANSCRIPT
![Page 1: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/1.jpg)
Introduction to Clustering
Dilan GörürUniversity of California, Irvine
June 2011iCAMP summer project
![Page 2: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/2.jpg)
Outline
• What is machine learning?
• What is clustering?
• Example application areas
• Challenges
• Approaches
‣ K-Means
‣ K-Medoids / Affinity propagation
‣ Linkage
• Implementation details
![Page 3: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/3.jpg)
Machine learning
• Machine learning provides methods that automatically learn from data.
• Extract the underlying structure in the data to summarize information.
• Accurately predict future data based on what we learn from current observations: generalization.
• Typically need a large collection of data or strong prior information.
![Page 4: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/4.jpg)
What is clustering?
• In classification, we have a labeled set of training points and we learn the grouping structure in a supervised way.
• Classification tries to predict the label of (unlabeled) data.
• Clustering is grouping things into “natural” categories when no class label is available
• Clustering is unsupervised learning.
• Need to automatically decide on the grouping structure.
![Page 5: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/5.jpg)
What is clustering?
• In classification, we have a labeled set of training points and we learn the grouping structure in a supervised way.
• Classification tries to predict the label of (unlabeled) data.
• Clustering is grouping things into “natural” categories when no class label is available
• Clustering is unsupervised learning.
• Need to automatically decide on the grouping structure.
![Page 6: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/6.jpg)
What is clustering?
• In classification, we have a labeled set of training points and we learn the grouping structure in a supervised way.
• Classification tries to predict the label of (unlabeled) data.
• Clustering is grouping things into “natural” categories when no class label is available
• Clustering is unsupervised learning.
• Need to automatically decide on the grouping structure.
![Page 7: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/7.jpg)
What is clustering?
• In classification, we have a labeled set of training points and we learn the grouping structure in a supervised way.
• Classification tries to predict the label of (unlabeled) data.
• Clustering is grouping things into “natural” categories when no class label is available
• Clustering is unsupervised learning.
• Need to automatically decide on the grouping structure.
![Page 8: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/8.jpg)
What is clustering?
• In classification, we have a labeled set of training points and we learn the grouping structure in a supervised way.
• Classification tries to predict the label of (unlabeled) data.
• Clustering is grouping things into “natural” categories when no class label is available
• Clustering is unsupervised learning.
• Need to automatically decide on the grouping structure.
![Page 9: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/9.jpg)
What is clustering?
• In classification, we have a labeled set of training points and we learn the grouping structure in a supervised way.
• Classification tries to predict the label of (unlabeled) data.
• Clustering is grouping things into “natural” categories when no class label is available
• Clustering is unsupervised learning.
• Need to automatically decide on the grouping structure.
![Page 10: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/10.jpg)
Why clustering?
• Labeling a large set of sample patterns can be costly.
• The contents of the database may not be known.
• Clustering can be used for finding features that will later be useful for categorization.
• It may help to gain insight into the nature of the data.
• It may lead to discovery of distinct subclasses or similarities among patterns.
![Page 11: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/11.jpg)
Why clustering?
• Labeling a large set of sample patterns can be costly.
• The contents of the database may not be known.
• Clustering can be used for finding features that will later be useful for categorization.
• It may help to gain insight into the nature of the data.
• It may lead to discovery of distinct subclasses or similarities among patterns.
![Page 12: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/12.jpg)
ClusteringGeneral approach for learning
• For a given set of points, learn a class assignment for each data point.
• Describe each cluster using a set of parameters .
• Use an objective (cost) function to measure the quality of clustering.
‣ A function of model parameters and assignment variables.
• Clustering is the process of optimizing the objective function.
xi, i ! {1, . . . , N}
c !c
yi ! {1, . . . ,K}
f(!, Y )
argmin f(!, Y )!, Y
![Page 13: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/13.jpg)
Clustering Challenges
• What cost function to use?
• What underlying structure to assume?
• How to measure similarity?
• How to decide on the number of clusters?
• Different answers to these questions may lead to different clustering algorithms and different clustering outcomes.
• Common objective: generalize well.
![Page 14: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/14.jpg)
Diffusion Tensor Imaging
• Cluster brain fiber tracts to remove clutter from images.
• Group anatomically similar fibers into bundles.
Klein et al., “Improving perception of brain structure using fiber clustering” SPIE Newsroom, 2007http://spie.org/documents/Newsroom/Imported/0771/0771-2007-07-25.pdf
![Page 15: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/15.jpg)
• Group related nodes together.
• Make bundles out of edges of the same cluster.
Software dependency structure
Telea and Ersoy, “Image-Based Edge Bundles: Simplified Visualization of LargeGraphs.” Computer Graphics Forum, 2010http://www.cs.rug.nl/~alext/PAPERS/EuroVis10/paper.pdf
![Page 16: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/16.jpg)
US migrations
Telea and Ersoy, “Image-Based Edge Bundles: Simplified Visualization of LargeGraphs.” Computer Graphics Forum, 2010http://www.cs.rug.nl/~alext/PAPERS/EuroVis10/paper.pdf
![Page 17: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/17.jpg)
Document browsing
• Search for relevant documents.
http://search.carrot2.org
![Page 18: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/18.jpg)
Topic analysis
Wang and McCallum. “Topics over Time: A Non-Markov Continuous-Time Model of Topical Trends, KDD 2006http://www.cs.umass.edu/~mccallum/papers/tot-kdd06.pdf
![Page 19: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/19.jpg)
Language families
![Page 20: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/20.jpg)
Spike sorting
Gorur et al. “Modelling spikes with mixtures of factor analysers.” DAGM, 2004
http://www.gatsby.ucl.ac.uk/~dilan/papers/pdf2646.pdf
![Page 21: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/21.jpg)
Categorizing images
1.6 1.1 0.5 0.0 1e−3
1.9 1.3 0.6 0.0 1e−3
1.7 1.1 0.6 0.0 1e−3
1.8 1.2 0.6 0.0 1e−3
1.4 1.0 0.5 0.0 1e−3
1.5 1.0 0.5 0.0 1e−3
![Page 22: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/22.jpg)
Discovering states of differentiation
Data from Cinquin Lab
![Page 23: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/23.jpg)
Many More Applications
• Marketing: finding groups of customers with similar behavior given a large database of customer data containing their properties and past buying records;
• Biology: classification of plants and animals given their features;
• Libraries: book ordering;
• Insurance: identifying groups of motor insurance policy holders with a high average claim cost; identifying frauds;
• City-planning: identifying groups of houses according to their house type, value and geographical location;
• Earthquake studies: clustering observed earthquake epicenters to identify dangerous zones;
• WWW: document classification; clustering weblog data to discover groups of similar access patterns.
![Page 24: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/24.jpg)
Why clustering?
• Labeling a large set of sample patterns can be costly.
• The contents of the database may not be known.
• Clustering can be used for finding features that will later be useful for categorization.
• It may help to gain insight into the nature of the data.
• It may lead to discovery of distinct subclasses or similarities among patterns.
![Page 25: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/25.jpg)
ClusteringGeneral approach for learning
• For a given set of points, learn a class assignment for each data point.
• Describe each cluster using a set of parameters .
• Use an objective (cost) function to measure the quality of clustering.
‣ A function of model parameters and assignment variables.
• Clustering is the process of optimizing the objective function.
xi, i ! {1, . . . , N}
c !c
yi ! {1, . . . ,K}
f(!, Y )
argmin f(!, Y )!, Y
![Page 26: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/26.jpg)
Clustering Challenges
• What cost function to use?
• What underlying structure to assume?
• How to measure similarity?
• How to decide on the number of clusters?
• Different answers to these questions lead to different clustering algorithms.
• Common objective: generalize well.
![Page 27: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/27.jpg)
Clustering Differences in algorithms
According to the resulting structure:
• Flat clustering.
• Hierarchical clustering.
According to the modeling framework:
• Deterministic clustering.
• Probabilistic models.
• Bayesian models.
![Page 28: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/28.jpg)
K-means
• A flat clustering technique.
• A prototype-based approach.
• The samples come from a known number of clusters with prototypes
• Each data point belongs to exactly one cluster.
• Alternative notation:
• Task: find good prototypes and and good assignments of data points to prototypes.
ric ={1 if xi belongs to cluster c,
0 otherwise.
yi ! {1, . . . ,K}
µc, c ! {1, . . . ,K}
![Page 29: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/29.jpg)
• Say we are given good prototypes.
• How should we assign data points to clusters?
• For each data point,
‣ Calculate the distance of the data point to each of the prototypes.
‣ Assign the point to its closest prototype.
K-meansAssigning data points to clusters
cyi = argmin ||xi ! µc||
![Page 30: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/30.jpg)
• Say we are given good prototypes.
• How should we assign data points to clusters?
• For each data point,
‣ Calculate the distance of the data point to each of the prototypes.
‣ Assign the point to its closest prototype.
K-meansAssigning data points to clusters
cyi = argmin ||xi ! µc||
![Page 31: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/31.jpg)
K-meansLearning good prototypes
• Say we have good assignments of data points to clusters.
• How can we find good prototypes?
• Define the prototypes to be the mean of each cluster.
µc =!N
i:yi=c xi
nc=
!Ni=1 ricxi!Ni=1 ric
![Page 32: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/32.jpg)
K-meansLearning good prototypes
• Say we have good assignments of data points to clusters.
• How can we find good prototypes?
• Define the prototypes to be the mean of each cluster.
µc =!N
i:yi=c xi
nc=
!Ni=1 ricxi!Ni=1 ric
![Page 33: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/33.jpg)
K-meansLearning good prototypes
• Say we have good assignments of data points to clusters.
• How can we find good prototypes?
• Define the prototypes to be the mean of each cluster.
µc =!N
i:yi=c xi
nc=
!Ni=1 ricxi!Ni=1 ric
![Page 34: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/34.jpg)
K-meansUpdates
• We neither have good assignments nor good prototypes to start with.
• Solution: iterate to find a good fit to both.
• Convergence: when the cluster assignments do not change anymore.
argminN!
i=1
K!
c=1
ric||xi ! µc||R,µ
![Page 35: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/35.jpg)
Implementation DetailsK-means
• Each data point belongs to one cluster only. Need assignment variables to remember the cluster membership of each data point.
• Need to remember the prototype vectors.
• Calculate distances between prototype vectors and data points.
• Find which prototype is the closest to each data point.
• Average over data points assigned to a cluster to find its prototype vector.
![Page 36: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/36.jpg)
Implementation DetailsK-means
• Initialize cluster assignments:
• Initialize prototype vectors:
• Repeat until convergence:
‣ Remember current assignments:
‣ Update cluster assignments:
- for
-
‣ Update prototype vectors:
-
• Converged if old assignments is the same as the new assignments.
r = ceil ( rand(1, N) + K)
µc = mean(X(Y == c, :))
oldY = Y
i = 1 : N
dd(k) =!
d
(xid ! µcd)2
[d1 r(i)] = min(dd)
µc = mean (X(Y == k, :))
![Page 37: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/37.jpg)
Affinity Propagation
• Similar idea to k-means:
‣ Find prototype vectors to represent clusters.
‣ Assign each data point to the cluster with closest prototype.
‣ Update prototype vectors.
• Difference: Prototypes are chosen from the dataset.
![Page 38: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/38.jpg)
Hierarchical Clustering
![Page 39: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/39.jpg)
• Shorter search time
• More intuitive relationships between groups of data points
• More structure in the representation, varying degrees of similarity
• Easier and nicer to visualize
Hierarchical clusteringMotivation
Uses of hierarchical clustering:
• recover underlying structure: we believe that the data has an underlying tree structure
• visualize data: understand relationship among data items
• summarize data: a way to partition data into different (possibly unrelated) groups
![Page 40: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/40.jpg)
• Top-down divisive.
‣ Start with assigning all data points to one (or a few coarse) cluster.
‣ Recursively split each cluster.
‣ Uses a flat clustering algorithm as a subroutine.
• Bottom-up agglomerative.
‣ Start with assigning each data point to its own cluster.
‣ Iteratively find pairs of clusters to merge.
‣ Clusters found by finding pairs with maximum similarity.
• Dominant approach is bottom-up: better search landscape, more flexible algorithms, but is more myopic.
Hierarchical clusteringDifferent approaches
![Page 41: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/41.jpg)
Hierarchical clusteringLinkage algorithm
• Choose a distance measure
‣ between pairs of points: e.g. Euclidian, Hamming
‣ between pairs of clusters of points: e.g. average linkage, minimum linkage
• Start with each data point in its own cluster at the leaf nodes
‣ compute distance of each pair of nodes
‣ merge the closest pair into a new node
‣ repeat
![Page 42: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/42.jpg)
Hierarchical clusteringLinkage algorithm
• Choose a distance measure
‣ between pairs of points: e.g. Euclidian, Hamming
‣ between pairs of clusters of points: e.g. average linkage, minimum linkage
• Start with each data point in its own cluster at the leaf nodes
‣ compute distance of each pair of nodes
‣ merge the closest pair into a new node
‣ repeat
![Page 43: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/43.jpg)
Hierarchical clusteringLinkage algorithm
• Choose a distance measure
‣ between pairs of points: e.g. Euclidian, Hamming
‣ between pairs of clusters of points: e.g. average linkage, minimum linkage
• Start with each data point in its own cluster at the leaf nodes
‣ compute distance of each pair of nodes
‣ merge the closest pair into a new node
‣ repeat
![Page 44: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/44.jpg)
Hierarchical clusteringLinkage algorithm
• Choose a distance measure
‣ between pairs of points: e.g. Euclidian, Hamming
‣ between pairs of clusters of points: e.g. average linkage, minimum linkage
• Start with each data point in its own cluster at the leaf nodes
‣ compute distance of each pair of nodes
‣ merge the closest pair into a new node
‣ repeat
![Page 45: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/45.jpg)
Hierarchical clusteringLinkage algorithm
• Choose a distance measure
‣ between pairs of points: e.g. Euclidian, Hamming
‣ between pairs of clusters of points: e.g. average linkage, minimum linkage
• Start with each data point in its own cluster at the leaf nodes
‣ compute distance of each pair of nodes
‣ merge the closest pair into a new node
‣ repeat
![Page 46: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/46.jpg)
Hierarchical clusteringLinkage algorithm
• Choose a distance measure
‣ between pairs of points: e.g. Euclidian, Hamming
‣ between pairs of clusters of points: e.g. average linkage, minimum linkage
• Start with each data point in its own cluster at the leaf nodes
‣ compute distance of each pair of nodes
‣ merge the closest pair into a new node
‣ repeat
![Page 47: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/47.jpg)
Hierarchical clusteringLinkage algorithm
• Choose a distance measure
‣ between pairs of points: e.g. Euclidian, Hamming
‣ between pairs of clusters of points: e.g. average linkage, minimum linkage
• Start with each data point in its own cluster at the leaf nodes
‣ compute distance of each pair of nodes
‣ merge the closest pair into a new node
‣ repeat
![Page 48: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/48.jpg)
Hierarchical clusteringLinkage algorithm
• Choose a distance measure
‣ between pairs of points: e.g. Euclidian, Hamming
‣ between pairs of clusters of points: e.g. average linkage, minimum linkage
• Start with each data point in its own cluster at the leaf nodes
‣ compute distance of each pair of nodes
‣ merge the closest pair into a new node
‣ repeat
![Page 49: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/49.jpg)
Hierarchical clusteringLinkage algorithm
• Choose a distance measure
‣ between pairs of points: e.g. Euclidian, Hamming
‣ between pairs of clusters of points: e.g. average linkage, minimum linkage
• Start with each data point in its own cluster at the leaf nodes
‣ compute distance of each pair of nodes
‣ merge the closest pair into a new node
‣ repeat
• Use distance in the representation.
![Page 50: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/50.jpg)
Linkage AlgorithmsDistance between clusters
• Single (or minimum) linkage:
• Complete (or maximum) linkage:
• Average linkage:
dist(A,B) = minxi!A,xj!B
dist(xi, xj)
dist(A,B) = maxxi!A,xj!B
dist(xi, xj)
dist(A,B) =1
|A||B|!
xi!A,xj!B
dist(xi, xj)
![Page 51: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/51.jpg)
Implementation DetailsLinkage
• Initialize a list of clusters with each data point in its own cluster.
• At each iteration, merge two clusters (nodes): there will be a total of N-1 iterations and 2N-1 nodes.
• Compute distances between every pair of nodes.
• Pick the pair of nodes with smallest distance.
• Merge the pair:
‣ Remove these nodes from representation.
‣ Include the new node in the representation.
• Repeat until a single node remains in the representation.
![Page 52: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/52.jpg)
Clustering Cell-Cycle Phase and Differentiation
![Page 53: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/53.jpg)
Cell-cycle phase and differentiation clusteringData and task description
Preprocessing
Automatic Seeding Segmentation
Original Image
![Page 54: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/54.jpg)
Cell-cycle phase and differentiation clusteringProcess details
Segmented Image Visualization
Feature Selection
Validation
Feature Extraction
Clustering
![Page 55: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/55.jpg)
Cell-cycle phase and differentiation clusteringResults
![Page 56: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/56.jpg)
Cell-cycle phase and differentiation clusteringResults
![Page 57: Introduction to Clustering - UCI Mathematics€¦ · ‣ Start with assigning all data points to one (or a few coarse) cluster. ‣ Recursively split each cluster. ‣ Uses a flat](https://reader035.vdocument.in/reader035/viewer/2022081405/5f076b447e708231d41ce3d9/html5/thumbnails/57.jpg)