part 4: advanced svm-based learning methods
Post on 11-Feb-2016
19 Views
Preview:
DESCRIPTION
TRANSCRIPT
1111
Part 4: ADVANCED SVM-based LEARNING METHODS
Electrical and Computer Engineering
Vladimir Cherkassky University of Minnesota
cherk001@umn.edu
Presented at Tech Tune Ups, ECE Dept, June 1, 2011
2
OUTLINE• Motivation for non-standard
approaches: high-dimensional data• Alternative Learning Settings
- Transduction and SSL- Inference Through Contradictions- Learning using privileged information (or SVM+)- Multi-task Learning
• Summary
3
Insights provided by SVM(VC-theory)• Why linear classifiers can generalize?
(1) Margin is large (relative to R)(2) % of SV’s is small(3) ratio d/n is small
• SVM offers an effective way to control complexity (via margin + kernel selection) i.e. implementing (1) or (2) or both
• What happens when d>>n ?- standard inductive methods usually fail
4
How to improve generalization for HDLSS?Conventional approach:
Incorporate a priori knowledge into learning method• Preprocessing and feature selection • Model parameterization (~ good kernels in SVM)Assumption: a priori knowledge about good model
Non-standard learning formulations:Incorporate a priori knowledge into new non-standard
learning formulation (learning setting)Assumption: a priori knowledge is about properties
of application data and/or goal of learning
• Which type of assumptions makes more sense?
5
OUTLINE
• Motivation for non-standard approaches
• Alternative Learning Settings- Transduction and SSL- Inference Through Contradictions- Learning with Structured Data- Multi-task Learning
• Summary
6
Examples of non-standard settings • Application domain: hand-written digit recognition• Standard inductive setting• Transduction: labeled training + unlabeled data• Learning through contradictions:
labeled training data ~ examples of digits 5 and 8unlabeled examples (Universum) ~ all other (eight) digits
• Learning using hidden information:Training data ~ t groups (i.e., from t different persons)Test data ~ group label not known
• Multi-task learning:Training data ~ t groups (from different persons)Test data ~ t groups (group label is known)
7
Modifications of Inductive Setting • Standard Inductive learning assumes
Finite training set Predictive model derived using only training dataPrediction for all possible test inputs
• Possible modifications1. Predict only for given test points transduction2. A priori knowledge in the form of additional ‘typical’ samples learning through contradiction3. Additional (group) info about training data Learning using privileged information (LUPI) aka SVM+4. Additional (group) info about training + test data Multi-task learning
ii y,x
8
Transduction (Vapnik, 1982, 1995)• How to incorporate unlabeled test data into the
learning process? Assume binary classification
• Estimating function at given pointsGiven: labeled training data and unlabeled test points
Estimate: class labels at these test points
Goal of learning: minimization of risk on the test set:
where
*jx mj ,...,1 ii y,x ni ,...,1
),....( **1
*myyy
)/(,1)( **
1
*jj
y
m
j
ydPyyLm
R xy
),(),....,( **1
* mff xxy
9
Induction vs Transduction
10
Transduction based on margin size
Single unlabeled test point X
11
Many test points X aka working samples
12
Transduction based on margin size • Binary classification, linear parameterization,
joint set of (training + working) samples
• Two objectives of transductive learning:(TL1) separate labeled training data using a large-margin hyperplane (as in standard inductive SVM)(TL2) separating (explain) working data set using a large-margin hyperplane.
13
Transduction based on margin size • Standard SVM hinge loss for labeled samples• Loss function for unlabeled samples:
Mathematical optimization formulation
14
Optimization formulation for SVM transduction• Given: joint set of (training + working) samples• Denote slack variables for training, for working • Minimize
subject to
where Solution (~ decision boundary)
• Unbalanced situation (small training/ large test) all unlabeled samples assigned to one class
• Additional constraint:
i*j
m
jj
n
ii CCbR
1
**
1
)(21),( www
mjnibyby
ji
jij
iii
,...,1,,...,1,0,1])[(1])[(
*
**
xwxw
mjbsigny jj ,...,1),(* xw** )()( bD xwx
m
ji
n
ii bm
yn 11
])[(11 xw
15
Optimization formulation (cont’d)• Hyperparameters control the trade-off
between explanation and margin size• Soft-margin inductive SVM is a special case of
soft-margin transduction with zero slacks• Dual + kernel version of SVM transduction• Transductive SVM optimization is not convex
(~ non-convexity of the loss for unlabeled data) – different opt. heuristics ~ different solutions
• Exact solution (via exhaustive search) possible for small number of test samples (m) – but this solution is NOT very useful (~ inductive SVM).
*CandC
0* j
16
Many applications for transduction• Text categorization: classify word documents
into a number of predetermined categories• Email classification: Spam vs non-spam• Web page classification• Image database classification• All these applications:
- high-dimensional data- small labeled training set (human-labeled)- large unlabeled test set
17
Example application
• Prediction of molecular bioactivity for drug discovery
• Training data~1,909; test~634 samples• Input space ~ 139,351-dimensional• Prediction accuracy:SVM induction ~74.5%; transduction ~ 82.3%Ref: J. Weston et al, KDD cup 2001 data analysis: prediction
of molecular bioactivity for drug design – binding to thrombin, Bioinformatics 2003
18
Semi-Supervised Learning (SSL)• Labeled data + unlabeled data Model• Similar to transduction (but not the same):
- Goal 1 ~ prediction for unlabeled samples - Goal 2 ~ estimate an inductive model
• Many algorithms• Applications similar to transduction• Typically
- Transduction works better for HDLSS- SSL works better for low-dimensional data
19
Example: Self-Learning Algorithm
Given initial labeled set L and unlabeled set U Repeat:(1) estimate a classifier using labeled set L(2) classify randomly chosen unlabeled sample
using decision rule estimated in Step (1)(3) move this new labeled sample to set LIterate steps (1) – (3) until all unlabeled
samples are classified.
20
Noisy Hyperbolas: unlabeled samples in greenInitial condition:
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9Iteration 1
Unlabeled SamplesClass +1Class -1
Example of Self-Learning Algorithm
21
Example of Self-Learning Algorithm
Iteration 50 Iteration 100 (final)
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9Iteration 50
Unlabeled SamplesClass +1Class -1
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9Iteration 100
Class +1Class -1
22
Inference through contradiction (Vapnik 2006)
• Motivation: what is a priori knowledge?- info about the space of admissible models- info about admissible data samples
• Labeled training samples + unlabeled samples from the Universum
• Universum samples encode info about the region of input space (where application data lives):- Usually from a different distribution than training/test data
• Examples of the Universum data• Large improvement for small training samples
23
Inference through contradictions aka Universum learning
24
Main Idea
Fig. courtesy of J. Weston (NEC Labs)
• Handwritten digit recognition: digit 5 vs 8
25
Learning with the Universum• Inductive setting for binary classification
Given: labeled training data and unlabeled Universum samples Goal of learning: minimization of prediction risk (as in standard inductive setting)
• Balance between two goals:- explain labeled training data using large-margin hyperplane- achieve maximum falsifiability ~ max # contradictions on the Universum
Math optimization formulation (extension of SVM)
*jx mj ,...,1
ii y,x ni ,...,1
26
-insensitive loss for Universum samples
x
y
*2
1
27
Random averaging Universum
Average
Class 1
Class -1 Hyper-plane
28
Random Averaging for digits 5 and 8• Two randomly selected examples
• Universum sample:
29
Application Study (Vapnik, 2006)• Binary classification of handwritten digits 5 and 8• For this binary classification problem, the following
Universum sets had been used:U1: randomly selected digits (0,1,2,3,4,6,7,9)
U2: randomly mixing pixels from images 5 and 8
U3: average of randomly selected examples of 5 and 8
Training set size tried: 250, 500, … 3,000 samples
Universum set size: 5,000 samples
• Prediction error: improved over standard SVM, i.e. for 500 training samples: 1.4% vs 2% (SVM)
30
Cultural Interpretation of Universum:jokes, absurd examples:
neither Hillary nor Obama dadaism
31
Application Study: predicting gender of human faces
• Binary classification setting• Difficult problem:
dimensionality ~ large (10K - 20K)labeled sample size ~ small (~ 10 - 20)
• Humans perform very well for this task• Issues:
- possible improvement (vs standard SVM)- how to choose ‘good’ Universum?- model parameter tuning
32
Male Faces: examples
33
Female Faces: examples
34
Universum Faces:neither male nor female
35
Empirical Study (cont’d)• Universum generation:
U1 Average: of male and female samples randomly selected from the training set (U. of Essex database)
U2 Empirical Distribution: estimate pixel-wise distribution of the training data. Generate a new picture from this distribution
U3 Animal faces:
3636
Universum generation: examples• U1 Averaging:
• U2 Empirical Distribution:
37
Results of gender classification
• Classification accuracy: improves vs standard SVM by ~ 2% with U1 Universum, and by ~ 1% with U2 Universum.
• Universum by averaging gives better results for this problem, when number of Universum samples N = 500 or 1,000
3838
Results of gender classification• Universum ~ Animal Faces:
• Degrades classification accuracy by 2-5% (vs standard SVM)
• Animal faces are not relevant to this problem
39
Learning with Structured Data(Vapnik, 2006)
• Application: Handwritten digit recognitionLabeled training data provided by t persons (t >1) Goal 1: find a classifier that will generalize well for future samples generated by these persons ~ Learning with Structured Data or Learning using Hidden InformationGoal 2: find t classifiers with generalization (for each person) ~ Multi-Task Learning (MTL)
• Application: Medical diagnosisLabeled training data provided by t groups of patients (t >1), say men and women (t = 2) Goal 1: estimate a classifier to predict/diagnose a disease using training data from t groups of patients ~ LWSDGoal 2: find t classifiers specialized for each group of patients ~ MTL
4040
Different Ways of Using Group Information
SVM
SVM+
SVM
SVM
svm+MTL
f(x)
f(x)
f1(x)
f2(x)
f1(x)
f2(x)
sSVM:
SVM+:
mSVM:
MTL:
41
SVM+ technology (Vapnik, 2006)• Map the input vectors simultaneously into:
- Decision space (standard SVM classifier)- Correcting space (where correcting functions model slack variables for different groups)
• Decision space/function ~ the same for all groups• Correcting functions ~ different for each group
(but correcting space may be the same)• SVM+ optimization formulation incorporates:
- the capacity of decision function- capacity of correcting functions for group r- relative importance (weight) of these two capacities
ww, rr ww ,
42
SVM+ approach (Vapnik, 2006)
Decision function
Decision space
Correcting space Correcting functions
Group1Group2Class 1Class -1
Correcting space
mapping
2
2
1
1
r slack variable for group r
mapping
43
SVM+ Formulation
r
tt Ti
ri
t
r
t
rrrddbwww
C 11,..,,,..,,
),(2
),(21 min
11
wwww
subject to:
trTiby rriii ,...,1,,1)),(( zw
trTi r
ri ,...,1,,0
trTid rrrri
ri ,...,1,,),( wz
]))(,[()]([ bsignfsigny Z xwxriz
r Zri
)(xz
Zizi )(xzix
Decision Space
Correcting Space
4444
SVM+ for Multi-task Learning (Liang 2008)
• New learning formulation: SVM+MTL• Define decision function for each group as
• Common decision function models the relatedness among groups
• Correcting functions fine-tune the model for each group (task)
.
trdbf rrzzr r,...,1,)),(()),(()( wxwxx
bz )),(( wx
4545
t
r Ti
ri
t
rrrddbwww
rrt
C11,..,,,..,,
),(2
),(21 min
11
wwww subject to:
trTidby rrir
riri
ri ,...,1,,1)),(),(( zwzw
trTiri ,...,1,,0
rizr Z
ri )(xz
Zizi )(xzix Correcting Space
svm+MTL FormulationDecision Space
trdbsignf rrzzr r,...,1,))),(()),((()( wxwxx
4646
Empirical Validation• Different ways of using group info
different learning settings:- which one yields better generalization?- how performance is affected by sample size?
• Empirical comparisons:- synthetic data set
4747
Different Ways of Using Group Information
SVM
SVM+
SVM
SVM
svm+MTL
f(x)
f(x)
f1(x)
f2(x)
f1(x)
f2(x)
sSVM:
SVM+:
mSVM:
MTL:
4848
Comparison for Synthetic Data Set
• Generate x where each • The coefficient vectors of three tasks are specified as
• For each task and each data vector, • Details of methods used:
- linear SVM classifier (single parameter C)- SVM+, SVM+MTL classifier (3 parameters: linear kernel for decision space, RBF kernel for correcting space, and parameter γ) - Independent validation set for model selection
20R 20,...,1,)1,1(~ i�uniformxi
)5.0( xisigny ]0,0,0,0,0,1,0,0,0,0,0,1,0,[]0,0,0,0,0,0,0,0,1,0,1,0,1,[
]0,0,0,0,0,0,0,0,0,0,1,1,1,[
3
2
1
1,11,1,1,1,1,β1,11,1,1,1,1,β1,11,1,1,1,1,β
4949
Experimental Results • Comparison results (ave over 10 trials):
n ~ number of training samples per taskave test error (%):
Note: relative performance depends on sample sizeNote: SVM+ always better than SVM
SVM+MTL always better than mSVM
Methods: sSVM SVM+ mSVM SVM+MTL
n=15 19.9 19.1 29.3 20.8n=100 11.9 11.7 8.8 8.5
50
OUTLINE
• Motivation for non-standard approaches• Alternative Learning Settings• Summary: Advantages/limitations of non-
standard settings
51
Advantages+limitations of nonstandard settings• Advantages
- make common sense- follow methodological framework (VC-theory)- yield better generalization (but not always)
• Limitations- need to formalize application requirements need to understand application domain- generally more complex learning formulations- more difficult model selection- few known empirical comparisons (to date)
• SVM+ is a promising new technology for hard problems
52
References and Resources• Vapnik, V. Estimation of Dependencies Based on Empirical Data. Empirical
Inference Science: Afterword of 2006, Springer, 2006 • Cherkassky, V. and F. Mulier, Learning from Data, second edition, Wiley, 2007• Chapelle, O., Schölkopf, B., and A. Zien, Eds., Semi-Supervised Learning, MIT
Press, 2006• Cherkassky, V. and Y. Ma, Introduction to Predictive learning, Springer, 2011
(to appear)• Hastie, T., R. Tibshirani and J. Friedman, The Elements of Statistical Learning.
Data Mining, Inference and Prediction, New York: Springer, 2001 • Schölkopf, B. and A. Smola, Learning with Kernels. MIT Press, 2002. Public-domain SVM software• Main web page link http://www.kernel-machines.org• LIBSVM software library http://www.csie.ntu.edu.tw/~cjlin/libsvm/• SVM-Light software library http://svmlight.joachims.org/• Non-standard SVM-based methodologies: Universum, SVM+, MTL
http://www.ece.umn.edu/users/cherkass/predictive_learning/
top related