andrew rosenberg- lecture 11: clustering introduction and projects machine learning

8/3/2019 Andrew Rosenberg- Lecture 11: Clustering Introduction and Projects Machine Learning

1/65

Lecture 11: Clustering Introduction and Projects

Machine Learning

Andrew Rosenberg

March 12, 2010

1 / 1
http://find/http://goback/


2/65

Last Time

Junction Tree Algorithm

Efficient Marginals in Graphical Models

2 / 1


3/65

Today

Clustering

Project Details

3 / 1


4/65

Clustering

Clustering

Clustering is an unsupervised Machine Learning application

The task is to group similar entities into groups.

4 / 1


5/65

We do this all the time

5 / 1


6/65


6 / 1

W d hi ll h i


7/65


7 / 1

W d hi i di i


8/65

We can do this in many dimensions

8 / 1

W d thi t d


9/65

We can do this to many degrees

9 / 1

W d thi i di i


10/65

We can do this in many dimensions

1 0 / 1

W d thi ll th ti


11/65


1 1 / 1

How do we set this up computationally?


12/65

How do we set this up computationally?

In Machine Learning, we optimize objective functions to find thebest solution.

Maximum Likelihood (for Frequentists)

Maximum A Posteriori (for Bayesians)

Empirical Risk Minimization

Loss function Minimization

What makes a good cluster?

How do we define loss or likelihood in a clustering solution?

1 2 / 1

Cluster Evaluation


13/65

Cluster Evaluation

Intrinsic Evaluation

Evaluate the compactness of the clusters

Extrinsic EvaluationCompare the results to some gold standard labeled data.(Not covered today)

1 3 / 1



14/65


Intercluster Variability (IV)

How different are the data points within the same cluster

Extracluster Variability (EV)

How different are the data points that are in distinct clusters

Minimize IV while maximizing EV.Minimize IVEV

IV =

C

xC

d(x, c)

d(x, c) = ||x c||

1 4 / 1

Degenerate Clustering Solutions


15/65


One Cluster

1 5 / 1



16/65


N Clusters

1 6 / 1

Clustering Approaches


17/65

Clustering Approaches

Hierarchical Clustering

Partitional Clustering

1 7 / 1



18/65


Recursive Partitioning

1 8 / 1



19/65



1 9 / 1



20/65



2 0 / 1



21/65



2 1 / 1



22/65

C g


2 2 / 1



23/65

g

Agglomerative Clustering

2 3 / 1



24/65

g


2 4 / 1



25/65

g


2 5 / 1



26/65

g


2 6 / 1



27/65

g


2 7 / 1



28/65


2 8 / 1



29/65


2 9 / 1



30/65


3 0 / 1



31/65


3 1 / 1



32/65


3 2 / 1



33/65


3 3 / 1

K-Means Clustering


34/65

K-Means clustering is a Partitional Clustering Algorithm.

Identify different partitions of the space for a fixed number ofclusters

Input: a value for K the number of clusters.

Output: the K centers of clusters centroids

3 4 / 1

K-Means Clustering


35/65

3 5 / 1

K-Means Clustering


36/65

Algorithm:

Given an integer K specifying the number of clusters.

Initialize K cluster centroids

Select K points from the data set at randomSelect K points from the space at random

For each point in the data set, assign it to the cluster whosecenter it is closest to.

argminCi d(x,Ci)

Update the centroid based on the points assigned to thecluster.

ci = 1|Ci|

xCi x

If any data point has changed clusters, repeat.

3 6 / 1

Why does K-Means Work?


37/65

When an assignment is changed, the sum of squared distancesof the data point to its assigned cluster is reduced.

IV is reduced.

When a cluster centroid is moved the sum of squareddistances of the data points within that cluster is reduced

IV is reduced.

At convergence we have found a local minimum of IV

3 7 / 1

K-Means Clustering


38/65

3 8 / 1

K-Means Clustering


39/65

3 9 / 1

K-Means Clustering


40/65

4 0 / 1

K-Means Clustering


41/65

4 1 / 1

K-Means Clustering


42/65

4 2 / 1

K-Means Clustering


43/65

4 3 / 1

K-Means Clustering


44/65

4 4 / 1

K-Means Clustering


45/65

4 5 / 1

K-Means Clustering


46/65

4 6 / 1

K-Means Clustering


47/65

4 7 / 1

K-Means Clustering


48/65

4 8 / 1

Soft K-Means

I K f d d i b h b f


49/65

In K-means, we forced every data point to be the member ofexactly one cluster.

We can relax this constraint.

p(x,Ci) =d(x, ci)jd(x, cj)

p(x,Ci) =

exp{d(x, ci)}

j exp{d(x, cj)}Based on minimizing entropy of cluster assignment.

We still define a cluster by a centroid, but we calculate thecentroid as a weighted center of all the data points.

ci =

x x p(x,Ci)xp(x,Ci)

Convergence is based on a stopping threshold rather thanchanging assignments.

4 9 / 1

Potential Problems with K-Means


50/65

Optimal?

K-means approaches a local minimum, but this is notguaranteed to be globally optimal.

Could you design an approach which is globally optimal?Consistent?

Different starting clusters can lead to different clustersolutions

5 0 / 1

Potential Problems with K-Means


51/65

Optimal?

K-means approaches a local minimum, but this is notguaranteed to be globally optimal.

Could you design an approach which is globally optimal?

Sure, in NP.

Consistent?

Different starting clusters can lead to different clustersolutions

5 1 / 1

Suboptimality in K-Means


52/65

5 2 / 1

Inconsistency in K-Means


53/65

5 3 / 1


54/65



55/65

5 5 / 1



56/65

5 6 / 1

More Clustering


57/65

K-Nearest Neighbors

Gaussian Mixture Models

Spectral Clustering

We will return to these.

5 7 / 1

The Project


58/65

Research Paper

Project

5 8 / 1

Research Paper


59/65

8-10 pages

Reporting on work in 4-5 papers.

Scope:One application areaor One technique

5 9 / 1

Research Paper: Application AreasIdentify an application that has made use of machine learning and


60/65

discuss how.

Graphics

Object Recognition

Optical Character Recognition

Superresolution

Segmentation

Natural Language Processing

Parsing

Sentiment Analysis

Information Extraction

Speech

Recognition

Synthesis

Discourse Analysis

Intonation

Game Playing

Scrabble

Craps

Prisoners Dilemma

Financials

Stock Prediction

Review Systems

Amazon

Netflix

Facebook

6 0 / 1

Research Paper: Techniques

Id if hi l i h i D ib i d


61/65

Identify a machine learning technique. Describe its use andvariants.

L1-regularization

Non-linear Kernels

Loopy Belief Propagation

Non-parametric BeliefPropagation

Soft-Decision Trees

Analysis of Neural NetworkHidden Layers

Structured Learning

Generalized Expectation

Evaluation Measures

Cluster EvaluationSemi-supervisedEvaluation

Graph Embedding

Dimensionality Reduction

Feature Selection

Graphical Model Construction

Non-parametric BayesianMethods

Latent Dirichlet Allocation

6 1 / 1

Project


62/65

Run a Machine Learning Experiment

Identify a problem/task data set.Implement one or more ML algorithmEvaluate the approach.

Write a Report of the Experiment

4 pages including references.Abstract 1 paragraph summarizing the experimentIntroduction describe the ProblemData Describe the data set, features extracted, etc.Method Describe the algorithm/approachResults Present and discuss resultsConclusion Summarize the experiment and results.

6 2 / 1

Project Ideas: TasksProjects can take any combination of Tasks and Approaches


63/65

GraphicsObject Classification

Facial RecognitionFingerprint IdentificationOptical Character RecognitionHandwriting recognition

(for languages/character systems other than English...)

LanguageTopic ClassificationSentiment AnalysisSpeech RecognitionSpeaker IdentificationPunctuation Restoration

Semantic SegmentationRecognition of Emotion, Sarcasm, etc.SMS Text normalizationChat participant identificationTwitter classification/threading

6 3 / 1


64/65

Games

ChessCheckersPoker (Poker Academy Pro)Blackjack

Recommenders (Collaborative Filtering)

NetflixCoursesJokesBooksFacebook?

Video Classification

Motion classificationSegmentation

6 4 / 1

Bye


65/65

Next

Hidden Markov Models

Viterbi Decoding

6 5 / 1

andrew rosenberg- lecture 11: clustering introduction and projects machine learning

Documents