fuzzy c-means clustering algorithm
TRANSCRIPT
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
1/34
1
Unsupervised Optimal Fuzzy Clustering
Presented by
Asya Nikitina
I.Gath and A. B. Geva. IEEE Transactions on Pattern
Analysis and Machine Intelligence,1989,11(7),773-781
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
2/34
2
Fuzzy Sets and Membership Functions
You are approaching a red light and must advise a drivingstudent when to apply the brakes. What would you say:
Begin braking 74 feetfrom the crosswalk?
Apply the brakespretty soon?
Everyday language is one example ofthe ways vagueness is
used and propagated.
Imprecision in data and information gathered from and about
our environment is eitherstatistical(e.g., the outcome ofacoin toss is a matter ofchance) ornonstatistical(e.g., apply
the brakes pretty soon).
This latter type ofuncertainty is calledfuzziness.
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
3/34
3
Fuzzy Sets and Membership Functions
We all assimilate and use fuzzy data, vague rules, and
imprecise information.
Accordingly, computational models ofreal systems shouldalso be able to recognize, represent, manipulate, interpret,
and use both fuzzy and statistical uncertainties.
Statistical models deal with random events and outcomes;
fuzzy models attempt to capture and quantify nonrandomimprecision.
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
4/34
4
Fuzzy Sets and Membership Functions
Conventional (or crisp) sets contain objects that satisfy
preciseproperties required for membership. For example, the
set ofnumbersHfrom 6 to 8 is crisp:
H= {r | 6 r8}
mH = 1; 6 r 8;
mH = 0; otherwise (mHis a membership function)
Crisp sets correspond to 2-valued logic: is orisnt
on oroff
blackorwhite
1 or0
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
5/34
5
Fuzzy Sets and Membership Functions
Fuzzy sets contain objects that satisfy impreciseproperties to
varying degrees, for example, the set ofnumbersFthat are
close to 7.
In the case offuzzy sets, the membership function, mF(r),maps numbers into the entire unit interval [0,1]. The value
mF(r)is called thegrade ofmembership ofrinF.
Fuzzy sets correspond to continuously-valued logic:
all shades ofgray between black(= 1) and white (= 0)
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
6/34
6
Fuzzy Sets and Membership Functions
Because the property close to 7 is fuzzy, there is not a
unique membership function forF. Rather, it is left to the
modeler to decide, based on the potential application and
properties desired forF, what mF(r)should be like.
The membership function is the basic idea in fuzzy set
theory; its values measure degrees to which objects satisfy
imprecisely defined properties.
Fuzzy memberships represent similarities ofobjects toimprecisely defined properties.
Membership values determine how much fuzziness a fuzzy
set contains.
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
7/34
7
Fuzziness and Probability
Pr(B ) = 0.91m(A) = 0.91
L = {all liquids} = fuzzy subset ofL: = {all potable liquids}
A B
swapmwa
t r? b r? H2O? HCl?
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
8/34
8
Clustering is a mathematical tool that
attempts to discover structures or
certain patterns in a data set, where
the objects inside each cluster showa certain degree ofsimilarity.
Clustering
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
9/34
9
Hard clusteringassign each feature
vector to one and only one oftheclusters with a degree ofmembership
equal to one and well defined
boundaries between clusters.
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
10/34
10
Fuzzy clusteringallows each feature
vector to belong to more than one
cluster with different membership
degrees (between 0 and 1) and
vague orfuzzy boundaries between
clusters.
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
11/34
11
Difficulties with Fuzzy Clustering
The optimal number ofclusters Kto becreated has to be determined (the
number ofclusters cannot always be
defined a priori and a good clustervalidity criterion has to be found).
The character and location ofcluster
prototypes (centers) is not necessarily
known a priori, and initial guesses
have to be made.
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
12/34
12
Difficulties with Fuzzy Clustering
The data characterized by large
variabilities in cluster shape, cluster
density, and the number ofpoints(feature vectors) in different clusters
have to be handled.
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
13/34
13
Objectives and Challenges
Create an algorithm forfuzzy clustering thatpartitions the data set into an optimal number
ofclusters.
This algorithm should account for variability
in cluster shapes, cluster densities, and the
number ofdata points in each ofthe subsets.
Cluster prototypes would be generated
through a process ofunsupervised learning.
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
14/34
14
The Fuzzy k-Means Algorithm
Nthe number offeature vectorsKthe number ofclusters (partitions)
q weighting exponent (fuzzifier; q > 1)
uikthe ith membership functionon the kth vector ( uik: X p [0,1] )
kuik= 1; 0 < iuik< nVithe cluster prototype (the mean ofall
feature vectors in clusteri or thecenter ofclusteri)
Jq(U,V)the objective function
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
15/34
15
Partition a set offeature vectors Xinto Kclusters (subgroups) represented as
fuzzy setsF1,F2, ,FK
by minimizing the objective functionJq(U,V)
Jq(U,V)= ik(uik)qd2(Xj Vi);KeN
Larger membership values indicate higher
confidence in the assignment ofthe pattern to
the cluster.
The Fuzzy k-Means Algorithm
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
16/34
16
Description of Fuzzy Partitioning
1) Choose primary cluster prototypes Vifor the values ofthe memberships
2) Compute the degree ofmembership of
all feature vectors in all clusters:
uij= [1/d2(Xj Vi)]
1/(q-1)/k[1/ d
2(Xj Vi)]1/(q-1) (1)
under the constraint: iuik= 1
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
17/34
17
Description of Fuzzy Partitioning
3) Compute new cluster prototypes Vi
Vi = j[(uij)qXj ] / j(uij)
q (2)
4) Iterate back and force between (1) and (2)
until the memberships or cluster centers
for successive iteration differ by more than
some prescribed value I (a termination
criterion)
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
18/34
18
The Fuzzy k-Means Algorithm
Computation ofthe degree ofmembership uijdependson the definition ofthe distance measure, d2(Xj Vi):
d2(Xj
Vi
) = (Xj
Vi
)T7 -1(Xj
Vi
)
7= I=>The distance is Euclidian, the shape ofthe
clusters assumed to be hyperspherical
7 is arbitrary =>The shape ofthe clusters assumed
to be ofarbitrary shape
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
19/34
19
The Fuzzy k-Means Algorithm
For the hyperellipsoidal clusters, an exponential
distance measure, d2e (Xj Vi), based on ML
estimation was defined:
d2e (Xj Vi) = [det(Fi)]1/2/Piexp[(Xj Vi)
TFi-1(Xj Vi)/2]
Fithe fuzzy covariance matrix ofthe ith cluster
Pi the a prioriprobability ofselecting ith cluster
h(i/Xj) = (1/d2e (Xj Vi))/k(1/d2e (Xj Vk))
h(i/Xj)the posterior probability (the probability of
selecting ith cluster given jth vector)
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
20/34
20
The Fuzzy k-Means Algorithm
Its easy to see that forq = 2, h(i/Xj) = u
ijThus, substituting uij with h(i/Xj)results in the fuzzy
modification ofthe ML estimation (FMLE).
Addition calculations for the FMLE:
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
21/34
21
The Major Advantage of FMLE
Obtaining good partition results starting fromgood classification prototypes.
The first layer ofthe algorithm, unsupervised
tracking ofinitial centroids, is based on the fuzzyK-means algorithm.
The next phase, the optimal fuzzy partition, is
being carried out with the FMLE algorithm.
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
22/34
22
Unsupervised Tracking of Cluster
Prototypes Different choices ofclassification prototypes
may lead to different partitions.
Given a partition into kcluster prototypes, place
the next (k+1)th cluster center in a region where
data points have low degree ofmembership in the
existing kclusters.
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
23/34
23
Unsupervised Tracking of Cluster
Prototypes
1) Compute average and standard deviation ofthe
whole data set.
2) Choose the first initial cluster prototype at the
average location ofall feature vectors.3) Choose an additional classification prototype
equally distant from all data points.
4) Calculate a new partition ofthe data set
according to steps 1) and 2) ofthe fuzzyk-means algorithm.
1) Ifk, the number ofclusters, is less than a given
maximum, go to step 3, otherwise stop.
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
24/34
24
Common Fuzzy Cluster Validity
Each data point has Kmemberships; so, it is
desirable to summarize the information by a
single number, which indicates how well the
data point (Xk) is classified by clustering.
i(uik)2 partition coefficient
i(uik) loguik classification entropy
maxi uik proportional coefficient
The cluster validity is just the average ofany
ofthose functions over the entire data set.
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
25/34
25
Proposed Performance Measures
Good clusters are actually not very fuzzy.
The criteria for the definition ofoptimal
partition ofthe data into subgroups were
based on the following requirements:
1. Clear separation between the resulting
clusters2. Minimal volume ofthe clusters
3. Maximal number ofdata points concentrated
in the vicinity ofthe cluster centroid
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
26/34
26
Proposed Performance Measures
Fuzzy hypervolume,FHV, is defined by:
WhereFi is given by:
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
27/34
27
Proposed Performance Measures
Average partition density,D
PA, is calculated from:
Where Si, the sum ofthe central members, is given by:
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
28/34
28
Proposed Performance Measures
The partition density,PD, is calculated from:
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
29/34
29
Sample Runs
In order to test the performance ofthe
algorithm, Nartificial m-dimensional
feature vectors from a multivariate normal
distribution having different parameters anddensities were generated.
Situations oflarge variability ofcluster
shapes, densities, and number ofdata points
in each cluster were simulated.
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
30/34
30
FCM Clustering with Varying Density
The higher density cluster attracts all other cluster prototypes
so that the prototype ofthe right cluster is slightly drawn away
from the original cluster center and the prototype ofthe left
cluster migrates completely into the dense cluster.
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
31/34
31
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
32/34
32
Fig. 3. Partition of12 clusters generated from five-
dimensional multivariate Gaussian distribution with
unequally variable features, variable densities and
variable number ofdata points ineach cluster (only threeofthe features are displayed).
(a) Data points before partitioning
(b) Partition of12 subgroups using the UFP-ONC algorithm.
All data points gave been classified correctly.
(a) (b)
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
33/34
33
-
8/8/2019 Fuzzy C-Means Clustering Algorithm
34/34
34
Conclusions
The new algorithm, UFP-ONC(unsupervised fuzzy partition-optimal number
ofclasses), that combines the most favorable
features ofboth the fuzzy K-means algorithm
and the FMLE, together with unsupervised
tracking ofclassification prototypes, were
created.
The algorithm performs extremely well insituations oflarge variability ofcluster shapes,
densities, and number ofdata points in each
cluster .