liang tian- learning from data through support vector machines

Learning From Data ThroughLearning From Data ThroughSupport Vector MachinesSupport Vector Machines

Liang Tian

tian@csee.wvu.edu

Lane Department of Computer Science& Electrical Engineering

West Virginia University

November 16, 2004November 16, 2004

CPE 520 CPE 520 –– Neural Networks Neural Networks

Neural Networks Neural Networks –– BP Learning BP Learning

P. Klinkhachorn. CpE520 Lecture Notes, CSEE Dept, West Virginia University.

BP Learning ProcedureBP Learning Procedure

P. Klinkhachorn. CpE520 Lecture Notes, CSEE Dept, West Virginia University.

ClassifierClassifier

A. Moore. Lecture Notes, School of Computer Science, CMU, http://www.cs.cmu.edu/~awm/tutorials.

ClassifierClassifier

SVM

Margin

MLP

ClassifierClassifier

ßß MLP MLP stop training when all points arestop training when all points arecorrectly classifiedcorrectly classified

ßß Decision surface may Decision surface may notnot be optimized be optimized

ßß Generalization error may Generalization error may notnot be minimized be minimized

Local MinimaLocal Minima

S. Bengio. An Introduction to Statistical Machine Learning – Neural Networks.IDIAP. Available at http://www.idiap.ch/~bengio May. 2003.

MLP MLP –– gradient descent learning gradient descent learning –– non linear optimization non linear optimization –– local minima local minima

SVM ClassificationSVM Classification

R. Collobert. An Introduction to Statistical Machine Learning – Support Vector Machines.IDIAP. Available at http://www.idiap.ch/~collober. Jan. 2003.

SVM ClassificationSVM Classification

R. Collobert. An Introduction to Statistical Machine Learning – Support Vector Machines.IDIAP. Available at http://www.idiap.ch/~collober. Jan. 2003.

SVM ClassificationSVM Classification

R. Collobert. An Introduction to Statistical Machine Learning – Support Vector Machines.IDIAP. Available at http://www.idiap.ch/~collober. Jan. 2003.

SVM ClassificationSVM Classification

R. Collobert. An Introduction to Statistical Machine Learning – Support Vector Machines.IDIAP. Available at http://www.idiap.ch/~collober. Jan. 2003.

Margin Maximization

Correct Classification

ßß Classic n Classic non-linear on-linear optimization optimization problemproblemwith inequality constraintswith inequality constraints

ßß Solved by maximizing the dual variables Solved by maximizing the dual variables

Lagrange functionLagrange function

SVM ClassificationSVM Classification

Subject to constraints

ßß Solutions of Lagrange Multipliers Solutions of Lagrange Multipliers ααii willwilldetermine parameters determine parameters ww and and bb

ßß The final decision The final decision hyperplane hyperplane is anis an

indicator functionindicator function

SVM ClassificationSVM Classification

Similar to weighted sum in MLP

SVM ClassificationSVM Classification

R. Collobert. An Introduction to Statistical Machine Learning – Support Vector Machines.IDIAP. Available at http://www.idiap.ch/~collober. Jan. 2003.

If it is non linear, itIf it is non linear, it’’s easier to separate two classes bys easier to separate two classes by

projecting data into projecting data into higher dimensionalhigher dimensional space. space.

SVM ClassificationSVM Classification

y = Outputy = S wi f(xi) +by = w fT(x)xi

InputSpace

x1x2

fi (x)

wi

b= w0 = bias

Z-Space

SVM ClassificationSVM Classification

Introducing Introducing KernelKernel Functions to simplify the computation Functions to simplify the computation

Problem?Computationally discouraging if dimensionality of Computationally discouraging if dimensionality of Z-SpaceZ-Space is very large is very large

Common Kernels areCommon Kernels are polynomial polynomial andand GaussianGaussian..

KernelKernel Function is in Function is in input spaceinput space

Bypass the high dimensionality of feature spaceBypass the high dimensionality of feature space

SVM Learning ExampleSVM Learning Example

Classic XOR Problem

SVM Learning ExampleSVM Learning Example

SVM Learning ExampleSVM Learning Example

SVM Learning ExampleSVM Learning Example

SVM Learning ExampleSVM Learning Example

SVM Learning ExampleSVM Learning Example

polynomial polynomial Kernel function KKernel function K

SVM Learning ProcedureSVM Learning Procedure

Step 1: Step 1: Select the kernel functionSelect the kernel function

Step 2: Step 2: Present inputs and desired outputsPresent inputs and desired outputs

Step 3: Step 3: SolveSolve Lagrange MultipliersLagrange Multipliers ααi i throughthrough an optimization probleman optimization problem

Step 4: Step 4: Obtain decision indicator functionObtain decision indicator function

BP Learning ProcedureBP Learning Procedure

P. Klinkhachorn. CpE520 Lecture Notes, CSEE Dept, West Virginia University.

SVM vs. NNSVM vs. NN

V. Kecman. Learning and Soft Computing. MIT Press, Cambridge, MA, 2001. ISBN: 0-262-11255-8.

This is a NNThis is a NN

SVM vs. NNSVM vs. NN

V. Kecman. Learning and Soft Computing. MIT Press, Cambridge, MA, 2001. ISBN: 0-262-11255-8.

This is a SVMThis is a SVM

SVM vs. NNSVM vs. NN

There is There is NO NO difference between structuredifference between structure

HOWEVERHOWEVER

ImportantImportant difference in difference in LEARNING !LEARNING !

[1] V. Vapnik. The Nature of Statistical Learning Theory. Springer, N.Y., 1995. ISBN: 0-387-94559-8.

ßß SVM is a novel type of machine learning SVM is a novel type of machine learningalgorithm developed by V. algorithm developed by V. VapnikVapnik..

ßß SVM minimizes an upper bound on the SVM minimizes an upper bound on thegeneralizationgeneralization error. error.

ßß Conventional neural networks only minimize Conventional neural networks only minimize

the error on the training data.the error on the training data.

ßß A unique and global solution and avoid being A unique and global solution and avoid beingtrapped at local minimatrapped at local minima

SVM vs. NNSVM vs. NN

SVM ApplicationsSVM Applications

Muller et al. “An introduction to kernel-based learning algorithms”, IEEE Trans. NN, 12(2), 2001, pp.181-201.

OCROCR

0.6%0.6%

SVM ApplicationsSVM Applications

Muller et al. “An introduction to kernel-based learning algorithms”, IEEE Trans. NN, 12(2), 2001, pp.181-201.

DNA Data AnalysisDNA Data Analysis

SVM ApplicationsSVM Applications

Tax and Duin, “Outliers and data descriptions”, Proceedings of the 7th Annual Conference of the Advanced School for Computing and Imaging, 2001. Pp. 234-241.

Single-Class ClassificationSingle-Class Classification

Two Types of ProblemsTwo Types of Problems

RegressionRegression ClassificationClassification

S. Bengio. An Introduction to Statistical Machine Learning – Neural Networks.IDIAP. Available at http://www.idiap.ch/~bengio May. 2003.

SVM RegressionSVM Regression

V. Kecman. Learning and Soft Computing. MIT Press, Cambridge, MA, 2001. ISBN: 0-262-11255-8.

SVM RegressionSVM Regression

Approximating the set of data of Approximating the set of data of ll pair of training patternpair of training pattern

The SVM model used for function approximation is:The SVM model used for function approximation is:

where where ΦΦ(x)(x) is the high-dimensional feature space is the high-dimensional feature space

that is nonlinearly mapped from the input space x.that is nonlinearly mapped from the input space x.

L. Tian and A. Noore, “A novel approach for short-term load forecasting using support vector machines,” International Journal of Neural Systems, vol. 14, no. 5, Oct. 2004.

SVM RegressionSVM Regression

ww and and bb can be estimated by minimizing the can be estimated by minimizing thefollowing regularized risk functionfollowing regularized risk function

Vapnik'sVapnik's linear loss function linear loss functionwith with εε-insensitivity zone-insensitivity zone

L. Tian and A. Noore, “A novel approach for short-term load forecasting using support vector machines,” International Journal of Neural Systems, vol. 14, no. 5, Oct. 2004.

SVM RegressionSVM Regression

is the weights vector norm, which is used tois the weights vector norm, which is used toconstrain the model structure capacity in orderconstrain the model structure capacity in orderto obtain better generalization performance.to obtain better generalization performance.

CC is the regularization constant, representing is the regularization constant, representingthe trade-off between the approximation errorthe trade-off between the approximation errorand the model structure.and the model structure.

L. Tian and A. Noore, “A novel approach for short-term load forecasting using support vector machines,” International Journal of Neural Systems, vol. 14, no. 5, Oct. 2004.

SVM RegressionSVM Regression

Minimizing risk objective function R Minimizing risk objective function R

L. Tian and A. Noore, “A novel approach for short-term load forecasting using support vector machines,” International Journal of Neural Systems, vol. 14, no. 5, Oct. 2004.

SVM RegressionSVM Regression

Then, the solution is given in the form: Then, the solution is given in the form:

L. Tian and A. Noore, “A novel approach for short-term load forecasting using support vector machines,” International Journal of Neural Systems, vol. 14, no. 5, Oct. 2004.

Training examples with (Training examples with (ααi i –– ααii*) *) ≠≠ 0 are 0 are support vectorssupport vectors

SVM RegressionSVM Regression

L. Tian and A. Noore, “A novel approach for short-term load forecasting using support vector machines,” International Journal of Neural Systems, vol. 14, no. 5, Oct. 2004.

Lagrange Multipliers can be obtained Lagrange Multipliers can be obtained by maximizing the form:by maximizing the form:

Regression Application - 1Regression Application - 1Short-Term Load ForecastingShort-Term Load Forecasting

L. Tian and A. Noore, “A novel approach for short-term load forecasting using support vector machines,” International Journal of Neural Systems, vol. 14, no. 5, Oct. 2004.

L. Tian and A. Noore, “A novel approach for short-term load forecasting using support vector machines,” International Journal of Neural Systems, vol. 14, no. 5, Oct. 2004.

Regression Application - 2Regression Application - 2Software Reliability PredictionSoftware Reliability Prediction

Regression Application - 2Regression Application - 2Software Reliability PredictionSoftware Reliability Prediction

Average Error = 1.20%Average Error = 1.20%

L. Tian and A. Noore, “On-line software reliability prediction: An approach based on support vector machines,” International Journal of Reliability, Quality and Safety Engineering, submitted and under revision.

L. Tian and A. Noore, “On-line software reliability prediction: An approach based on support vector machines,” International Journal of Reliability, Quality and Safety Engineering, submitted and under revision.

Regression Application - 2Regression Application - 2Software Reliability PredictionSoftware Reliability Prediction

Parameter SelectionParameter Selection

Cao and Tay, “Support vector machine with adaptive parameters in financial time series forecasting”, IEEE Trans. NN, 14(6), Nov. 2003, pp. 1506-1518.

ßß Both NN and SVM learn from experimental data Both NN and SVM learn from experimental data

ßß Both NN and SVM are universal Both NN and SVM are universal approximatorapproximator

ßß After learning, both NN and SVM have sameAfter learning, both NN and SVM have same

mathematical model, graphical representationmathematical model, graphical representation

ßß The only The only differencedifference is the is the learning methodlearning method..

ßß NN NN –– gradient descent gradient descent

ßß SVM SVM –– solving quadratic programming solving quadratic programming

SummarySummary

SVM Research IssuesSVM Research Issues

ßß Speeding up learning time when data is large. Speeding up learning time when data is large.

ßß Chunking, using subset of data Chunking, using subset of data

ßß Optimization techniques improvement Optimization techniques improvement

ßß Parameter selection and optimization Parameter selection and optimization

ßß Modified and adaptive SVM and some variations Modified and adaptive SVM and some variations

References and Further ReadingReferences and Further Reading

[1] V. Vapnik.The Nature of Statistical Learning Theory.Springer, N.Y., 1995. ISBN: 0-387-94559-8.

[2] S. Bengio.An Introduction to Statistical Machine Learning –Neural Networks.IDIAP. Available at http://www.idiap.ch/~bengioMay. 2003.

[3] V. Kecman.Learning and Soft Computing.MIT Press, Cambridge, MA, 2001. ISBN: 0-262-11255-8.

References and Further ReadingReferences and Further Reading

[4] R. Collobert.An Introduction to Statistical Machine Learning –Support Vector MachinesIDIAP. Available at http://www.idiap.ch/~colloberJan. 2003.

[5] L. Tian, A. Noore.A Novel Approach for Short-Term Load ForecastingUsing Support Vector Machines.International Journal of Neural Systems.International Journal of Neural Systems.Vol. 14, No. 5, Oct. 2004.

[6] L. Tian, A. Noore.On-line Software Reliability Prediction: An ApproachBased on Support Vector Machines.International Journal of Reliability, Quality andInternational Journal of Reliability, Quality andSafety Engineering.Safety Engineering.Submitted and under revision.

References and Further ReadingReferences and Further Reading

[7] V. Vapnik.Statistical Learning Theory.John Wiley & Sons, 1998, ISBN: 0-471-03003-1.

[8] http://www.kernel-machines

[9] http://www.support-vector.ws

Thank You !!Thank You !!

Questions and Comments ?Questions and Comments ?