feature extraction using fuzzy complete linear discriminant analysis the reporter : cui yan 2012....
TRANSCRIPT
Feature extraction using fuzzy complete linear discriminant analysis
The reporter : Cui Yan
2012. 4. 26
The report outlines
1.The fuzzy K-nearest neighbor classifier (FKNN)
2.The fuzzy complete linear discriminant analysis
3.Expriments
The Fuzzy K-nearest neighbor classifier (FKNN)
Each sample should be classified
similarly to its surrounding samples,
therefore, a unknown sample could be
predicated by considering the
classification of its nearest neighbor
samples.
The K-nearest neighbor classifier (KNN)
KNN tries to classify an unknown sample based
on its k-known classification neighbors.
FKNN
Given a sample set , a fuzzy M
-class partition of these vectors specify the membership degrees of each sample corres-ponding to each class.
The membership degree of a training vector to o each of M classes is specified by ,
which is computed by the following steps:ijuijx
},,,{ 21 nxxxX
Step 1: Compute the distance matrix between
pairs of feature vectors in the training.
Step 2: Set diagonal elements of this matrix to
infinity (practically place large numeric
values there).
Step 3: Sort the distance matrix (treat each of
its column separately) in an ascending
order. Collect the class labels of the
patterns located in the closest neigh-
borhood of the pattern under consi-
deration (as we are concerned with k
neighbors, this returns a list of k
integers).
Step 4: Compute the membership grade to class i
for j-th pattern using the expression proposed
in [1].
[1] J.M. Keller, M.R. Gray, J.A. Givens, A fuzzy k-nearest neighbor algorithm, IEEE Trans. Syst.Man Cybernet. 1985, 15(4):580-585
pattern. theof label the if )(*0.49
pattern. theof label the if )(*49.051.0
j-thi
j-thiu
kn
kn
ijij
ij
A example for FKNN
No. Feature1 Feature2 class
1 0.2000 0.3000 1
2 0.3000 0.2000 1
3 0.4000 0.3000 1
4 0.5000 0.5000 2
5 0.6000 0.4000 2
6 0.5000 0.6000 2
7 0.7000 0.3000 3
8 0.8000 0.4000 3
9 0.7000 0.5000 3
1 2 3 4 5 6 7 8 9
0
0.1414
0.2000
0.3606
0.4123
0.4243
0.5000
0.5385
0.6083
0
0.1414
0.1414
0.3606
0.3606
0.4123
0.4472
0.5000
0.5385
0
0.1414
0.2000
0.2236
0.2236
0.3000
0.3162
0.3606
0.4123
0
0.1000
0.1414
0.2000
0.2236
0.2828
0.3162
0.3606
0.3606
0
0.1414
0.1414
0.1414
0.2000
0.2236
0.2236
0.3606
0.4123
0
0.1000
0.2236
0.2236
0.3162
0.3606
0.3606
0.4243
0.4472
0
0.1414
0.1414
0.2000
0.2828
0.3000
0.3606
0.4123
0.5000
0
0.1414
0.1414
0.2000
0.3162
0.3606
0.4123
0.5385
0.6083
0
0.1414
0.1414
0.2000
0.2000
0.2236
0.3606
0.5000
0.5385
No. 1 2 3 4 5 6 7 8 9
1 0 0.1414 0.2000 0.3606 0.4123 0.4243 0.5000 0.6083 0.5385
2 0.1414 0 0.1414 0.3606 0.3606 0.4472 0.4123 0.5385 0.5000
3 0.2000 0.1414 0 0.2236 0.2236 0.3162 0.3000 0.4123 0.3606
4 0.3606 0.3606 0.2236 0 0.1414 0.1000 0.2828 0.3162 0.2000
5 0.4123 0.3606 0.2236 0.1414 0 0.2236 0.1414 0.2000 0.1414
6 0.4243 0.4472 0.3162 0.1000 0.2236 0 0.3606 0.3606 0.2236
7 0.5000 0.4123 0.3000 0.2828 0.1414 0.3606 0 0.1414 0.2000
8 0.6083 0.5385 0.4123 0.3162 0.2000 0.3606 0.1414 0 0.1414
9 0.5385 0.5000 0.3606 0.2000 0.1414 0.2236 0.2000 0.1414 0
1 2 3 4 5 6 7 8 9
1 2 3 4 5 6 7 8 9
2 1 2 6 4 4 5 9 5
3 3 1 5 9 5 8 7 8
4 4 5 9 7 9 9 5 4
5 5 4 3 8 3 4 4 7
6 7 7 7 6 7 3 6 6
7 6 6 8 3
9 9 9 1 2
8
1
6
2
3
2
3
2
8 8 8 2 1 2 1 1 1
1 2 3 4 5 6 7 8 9
1 1 1 2 2 2 3 3 3
1 1 1 2 2 2 2 3 2
1 1 1 2 3 2 3 3 3
2 2 2 3 3 3 3 2 2
2 2 2 1 3 1 2 2 3
2 3 3 3 2 3 1 2 2
3 2 2 3 1
3 3 3 1 1
3
1
2
1
1
1
1
1
3 3 3 1 1 1 1 1 1
1 2 3 4 5 6 7 8 9
1 1 1 2 2 2 2 3 2
1 1 1 2 3 2 3 3 3
2 2 2 3 3 3 3 2 2
Set k=3
1 2 3 4 5 6 7 8 9
0.8367 0.8367 0.8367 0 0 0 0 0 0
0.1633 0.1633 0.1633 0.8367 0.6733 0.8367 0.1633 0.1633 0.3267
0 0 0 0.1633 0.3267 0.1633 0.8367 0.8367 0.6733
The fuzzy complete linear
discriminant analysis
For the training set , we define the i-th class mean by combining the fuzzy membership degree as
And the total mean as
},,,{ 21 nxxxX
.,,2,1 ,1
1 ciu
xum n
j ij
n
j jij
i
1
1
n
i in xm
(1)
(2)
Incorporating the fuzzy membership degree, the between-class, the within-class and the total class fuzzy scatter matrix of samples can be defined as
c
i Nj
TjjijtF
c
i Nj
TijijijwF
c
i
n
j
TiiijbF
i
i
mxmxuS
mxmxuS
mmmmuS
1
1
1 1
))((
))((
))((
(3)
step1: Calculate the membership degree matrix U
by the FKNN algorithm.
step 2: According toEqs.(1)-(3) work out the
between-class, within-class and total class
fuzzy scatter matrices.
step 3: Work out the orthogonal eigenvectors
p1, . . . , pl of the total class fuzzy scatter
matrix corresponding to positive
eigenvalues.
Algorithm of the fuzzy complete linear analysis
tFS
step 4: Let P = (p1, . . . , pl) and
, work out the orthogonal
eigenvectors g1, . . . , gr of correspending
the zero eigenvalues.
step 5: Let P1 = (g1, . . . , gr) and , work
out the orthogonal eigenvectors v1, . . . , vr of
, calculate the irregular discriminant
vectors by .
P, S P S wFT
wF ˆ
P SP S bFT
bF ˆ
1ˆ1~
PS PS bFT
bF
bFS~
vPPwir 1irw
wFS
step 6: Work out the orthogonal eigenvectors q1,…, qs
of correspending the non-zero eigenvalues.
step 7: Let P2 = (q1,…, qs) and
, work out the optimal
discriminant vectors vr+1, . . . , vr+s by the
Fisher LDA, calculate the regular discriminant
vectors by .
step 8: (Recognition): Project all samples into the
obtained optimal discriminant vectors and
classify.
wFS
,PS P S wFT
wF 2ˆ2
2ˆ2 PS P S bFT
bF
rw vPPwr 2
Experiments
• We compare Fuzzy-CLDA with CLDA, UWLDA, FLDA, Fuzzy Fisherface, FIFDA on 3 different data sets from the UCI data sources. The characteristics of the three datasets can be found from (http://archive.ics.uci.edu/ml/datasets).
• All data sets are randomly split to the train set and test set with the ratio 1:4. Experiments are repeated 25 times to obtain mean prediction error rate as a performance measure, NCC is adopted to classify the test samples by using L2 norm.
Thanks!
2012. 4. 26