Beyond One-Class Classification

Amfeng

24 March 2009

Outline

Two models for One Class ClassificationFrom One Class to Binary ClassFrom Binary Class to Multi ClassFrom Multi Class to ClusteringConclustion

Two models for One Class Classification

One Class SVM Find the optimal hyperplane to separate the tar

get class from the origin with maximum margin

Support Vector Data Description Use the minimum hyperspere to enclose the tar

get class

Interpretation of the above models

（ a ） OCSVM 取高斯核时的最优超平面 （ b ） SVDD 取高斯核时的最小超球

How to extend to Binary or Multi Classification

For imbalance data

From SVDD to Binary SVDD with negative data: B_SVDD_Neg

Objective function:

21 2

2 2

2 2

( , , )

|| ||. . , 0, 0,

|| ||

target class, class

i pi p

i ii p

p p

R a R C C

x a Rs t

x a R

i p negative

Drawback: without considering the margin between classes.

The other Version of B_SVDD_Neg

Dong, X., W. Zhaohui, et al. (2001). A new multi-class support vector machines, Systems, Man, and Cybernetics, 2001 IEEE International Conference on.

Embedding margin for B_SVDD_Neg

范炜 (2003). 支持向量机算法的研究及其应用 , 浙江大学 . PhD.

The dual form

Notice

How to get the radius R

Must find the support vector on the hypersphere that negative live

d

Does it work really?

1

1

1

0

to KKT, if >0 =0

then 1

n

ii

n

ii

here

according

No support vector of

Negative data.Can’t calculate R

Solutions for above problem

1. Modify the coefficient of R: Biased support vector machine

Chan, C.-H., K. Huang, and M.R.L.a.I. King. Biased support vector machine for relevance feedback in image retrieval. in International Joint Conference on Neural Networks 2004. Budapest, Hungary.

In order to avoid the above problem, b need to less than 1, that is 1b

Equivalent style: Minimum Enclosing and Maximum Excluding Machine

Liu, Y. and Y.F. Zheng. Minimum Enclosing and Maximum Excluding Machine for Pattern Description and Discrimination Pattern Recognition. in Proc of the 18th Int Conf on ICPR 2006.Loa Alamitos: IEEE Computer Society

2. Modify the coefficient of margin

Here,

Wang, J., N. P, et al. (2005). Pattern classification via single spheres, Lecture notes in artificial intelligence.( briefly PCSS)

, 1here K

3. Modify the coefficients of margin and R

张新峰 ; 刘垚巍 : 广义超球面 SVM研究 , 计算机研究与发展 2008.1

1

Generalized HyperSphere SVM(GHSSVM)

Extend to Ellipsoid

Wei2007:Minimum Mahalanobis Enclosing Ellipsoid Machine for Pattern Classification:ICIC 2007,CCIS2, pp. 1176-1185

SVDD with negative data for Multi-Class:M_SVDD_Neg

Drawback: without considering the margin either .

Embedding margin for SVDD_Mulit: MSM_SVM

Pei-Yi Hao, Jung Hsien Chiang, Yen Hsiu lin:A new maximal-margin spherical-structured multi-class support vector machine, Appl Intell, 2009,30,P98-111

Dual formulation

Without the problem discussed at the

former.

Illustration of the difference

How about the hypenplane model

OCSVM with negative: Binary OCSVM_Neg

Motivation: using the mean of the other class instead of the optimal point. Doesn’t considering margin either.

1

1 1min

2

1. . ( ) , 0, 1,...,

n

ii

i n i i

n

s t x z i nt

Tw,ξ,

T

w w

w

From OCSVM to Asymmetric SVM: margin embededLike the SVDD Multi with margin, here also describe the target class by core hyperplane, then push the negative class by maximized the margin.

S. H. Wu, K. P. Lin, C. M. Chen, M. S. Chen, Asymmetric support vector machines: low false positive learning under the user tolerance, Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, 749-757, 2008.

Summarize

Model Hyperspere Ellipsoid Hyperplane

One-Class SVDD  MVEE ,MVCE, MELM

OCSVM

Binary-Class Without margin

B_SVDD_Neg B_OCSVM_Neg

Embedding margin

BSVM, MEMEMPCSS, GHSSVM

  Binary MELM

ASVM

Multi-Class Without margin

Multi SVDD_Neg

  One against others Or One-to One

?Embedding margin

MSM SVM  ?

One Class Classification for Clustering

Support Vector Clustering(JMLR2002)Iterative strategy integrating two-stage

one-class SVMKernel Growth (PAMI 2005)Soft Clustering for Kernel Growth

Support Vector Clustering

Clustering boundary: same as SVDD, found the support vector to get the boundary.

Clustering number: based on the adjacency matrix which components decided according to:

Ben-Hur, H. A., D., et al. (2002). "Support vector clustering

" Journal of Machine Learning Research 2 125-137.

The kernel width decided the clustering number

Outlier enable makes the clustering possible

Iterative strategy integrating two-stage one-class SVM

Yeh, C.-Y. and S.-J. Lee (2007). A Kernel-Based Two-Stage One-Class Support Vector Machines Algorithm. Advances in Neural Networks – ISNN 2007.

Different from SVC, need to know the clustering number in advance, attribute to partition-based clustering algorithm.

First stage: using OCSVM for each cluster to find the non-support vectors ;

Second stage: retrain the OCSVM using those non-support vector for representing each clustering accurately by the optimal hyperplane.

Illustration

Clustering assignment: each pattern is assign to the maximum projection value by:

Conclusion

One Class Classifier of SVDD and OCSVM can be used in many field, including:

Binary/Multi Class for unbalance data Clustering Large scale problem: CVM &BVM De-noising Information processing

Document classification Image retrieval ….