k-means and gaussian mixture model 王养浩 2013 年 11 月 20 日
TRANSCRIPT
SDP-MARCH-Talk
K-means and Gaussian Mixture Model
王养浩2013年 11月 20日
Outline
• K-means• Gaussian Mixture Model• Expectation Maximum
K-means
• Gather data points to a few cohesive ‘Clusters’
• Unsupervised Learning
K-means
K-means
K-means
• Easy• Fast
• Euclidean distance?• K needs input?• Convergence?
Determination of K
• Rule of Thumb:• Elbow Method• Cross Validation
2/nk
K-means Convergence
• x(i) data points • μc(i) cluster centroids
• Coordinate descent
Coordinate Descent
K-means Convergence
• Local minimum– The optimization object is non-convex
Gaussian Mixture Model
• Mixture of Gaussian distribution
Gaussian Mixture Model
• Log likelihood
• Maximum likelihood – Expectation Maximum
Expectation Maximum
Expectation Maximum
• Jenson inquality
Expectation Maximum
• Training set {}• Hidden variables {}• Parameter θ
Expectation Maximum
• Construct lower bound
Expectation Maximum
• Maximum lower bound– Coordinate Ascent on J
• Repeat until convergence –Maximum given fixed θ–Maximum θ given fixed
Expectation Maximum
• Repeat until convergence
Generalized Expectation Maximum
• Difficulty in M-step
Summary
• K-means– Coordinate descent
• Gaussian Mixture Model– Expectation Maximum
• Expectation Maximum–MLE for models with latent variables– Generalized EM
• Thanks!