support vector regression with r - svm tutorial
DESCRIPTION
SVM in RTRANSCRIPT
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 1/19
SVM Tutorial
Support Vector Regression with RIn this article I will show how to use R to perform a Support Vector Regression.
We will 䁀䁀rst do a simple linear regression, then move to the Support Vector Regression so that you
can see how the two behave with the same data.
A simple data set
To begin with we will use this simple data set:
I just put some data in excel. I prefer that over using an existing well-known data-set because the
purpose of the article is not about the data, but more about the models we will use.
As you can see there seems to be some kind of relation between our two variables X and Y, and it
look like we could 䁀䁀t a line which would pass near each point.
Let's do that in R !
Step 1: Simple linear regression in R
Here is the same data in CSV format, I saved it in a 䁀䁀le regression.csv :
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 2/19
We can now use R to display the data and 䁀䁀t a line:
The code above displays the following graph:
010203040506070809101112
# Load the data from the csv filedataDirectory <‐ "D:/" # put your own folder heredata <‐ read.csv(paste(dataDirectory, 'regression.csv', sep=""), header = TRUE) # Plot the dataplot(data, pch=16) # Create a linear regression modelmodel <‐ lm(Y ~ X, data) # Add the fitted lineabline(model)
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 3/19
Step 2: How good is our regression ?
In order to be able to compare the linear regression with the support vector regression we 䁀䁀rst need
a way to measure how good it is.
To do that we will change a little bit our code to visualize each prediction made by our model
This produces the following graph:
0102030405060708091011
dataDirectory <‐ "D:/"data <‐ read.csv(paste(dataDirectory, 'regression.csv', sep=""), header = TRUE) plot(data, pch=16)model <‐ lm(Y ~ X , data) # make a prediction for each XpredictedY <‐ predict(model, data) # display the predictionspoints(data$X, predictedY, col = "blue", pch=4)
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 4/19
For each data point the model makes a prediction displayed as a blue cross on the graph.
The only di၀倀erence with the previous graph is that the dots are not connected with each other.
In order to measure how good our model is we will compute how much errors it makes.
We can compare each value with the associated predicted value and see how far away they
are with a simple di၀倀erence.
Note that the expression is the error, if we make a perfect prediction will be equal to
and the error will be zero.
If we do this for each data point and sum the error we will have the sum of the errors, and if we
takes the mean we will get the Mean Squared Error (MSE)
A common way to measure error in machine learning is to use the Root Mean Squared Error
(RMSE) so we will use it instead.
To compute the RMSE we take the square root and we get the RMSE
Xi Y i
Yi Y i
−Y i Yi Y i Yi
MSE = ( −1n
∑i=1
n
Y i Yi)2
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 5/19
Using R we can come with the following code to compute the RMSE
We know now that the RMSE of our linear regression model is 5.70. Let's try to improve it with SVR !
Step 3: Support Vector Regression
In order to create a SVR model with R you will need the package e1071. So be sure to install it and to
add the library(e1071) line at the start of your 䁀䁀le.
Below is the code to make predictions with Support Vector Regression:
As you can see it looks a lot like the linear regression code. Note that we called the svm function (not
svr !) it's because this function can also be used to make classi䁀䁀cations with Support Vector
Machine. The function will automatically choose SVM if it detects that the data is categorical (if the
variable is a factor in R).
The code draws the following graph:
RMSE = MSE− −−−−
√
1234567
rmse <‐ function(error){ sqrt(mean(error^2))} error <‐ model$residuals # same as data$Y ‐ predictedYpredictionRMSE <‐ rmse(error) # 5.703778
12345
model <‐ svm(Y ~ X , data) predictedY <‐ predict(model, data) points(data$X, predictedY, col = "red", pch=4)
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 6/19
This time the predictions is closer to the real values ! Let's compute the RMSE of our support vector
regression model.
As expected the RMSE is better, it is now 3.15 compared to 5.70 before.
But can we do better ?
Step 4: Tuning your support vector regression model
In order to improve the performance of the support vector regression we will need to select the best
parameters for the model.
In our previous example, we performed an epsilon-regression, we did not set any value for epsilon ( ), but it took a default value of 0.1. There is also a cost parameter which we can change to avoid
over䁀䁀tting.
The process of choosing these parameters is called hyperparameter optimization, or model
1234
# /!\ this time svrModel$residuals is not the same as data$Y ‐ predictedY# so we compute the error like thiserror <‐ data$Y ‐ predictedYsvrPredictionRMSE <‐ rmse(error) # 3.157061
ϵ
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 7/19
selection.
The standard way of doing it is by doing a grid search. It means we will train a lot of models for the
di၀倀erent couples of and cost, and choose the best one.
There is two important points in the code above:
we use the tune method to train models with and cost =
which means it will train 88 models (it can take a long time)
the tuneResult returns the MSE, don't forget to convert it to RMSE before comparing the value
to our previous model.
The last line plot the result of the grid search:
ϵ
12345678
# perform a grid searchtuneResult <‐ tune(svm, Y ~ X, data = data, ranges = list(epsilon = seq(0,1,0.1), cost = 2^(2:9)))print(tuneResult)# best performance: MSE = 8.371412, RMSE = 2.89 epsilon 1e‐04 cost 4# Draw the tuning graphplot(tuneResult)
ϵ = 0,0.1,0.2, . . . , 1, , , . . . ,22 23 24 29
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 8/19
On this graph we can see that the darker the region is the better our model is (because the RMSE
is closer to zero in darker regions).
This means we can try another grid search in a narrower range we will try with values between 0
and 0.2. It does not look like the cost value is having an e၀倀ect for the moment so we will keep it as it
is to see if it changes.
We trained di၀倀erent 168 models with this small piece of code.
As we zoomed-in inside the dark region we can see that there is several darker patch.
From the graph you can see that models with C between 200 and 300 and between 0.8 and 0.9
have less error.
Hopefully for us, we don't have to select the best model with our eyes and R allows us to get it very
easily and use it to make predictions.
ϵ
123456
tuneResult <‐ tune(svm, Y ~ X, data = data, ranges = list(epsilon = seq(0,0.2,0.01), cost = 2^(2:9))) print(tuneResult)plot(tuneResult)
ϵ
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 9/19
We improved again the RMSE of our support vector regression model !
If we want we can visualize both our models. The 䁀䁀rst SVR model is in red, and the tuned SVR model
is in blue on the graph below :
I hope you enjoyed this introduction on Support Vector Regression with R.
You can download the source code of this tutorial. Each step has its own 䁀䁀le.
If you have any question feel free to comment.
Alexandre KOWALCZYKI am passionate about machine learning and Support Vector Machine. When I am not writing thisblog, you can �赯nd me on Kaggle participating in some competition.
12345678
tunedModel <‐ tuneResult$best.modeltunedModelY <‐ predict(tunedModel, data) error <‐ data$Y ‐ tunedModelY # this value can be different on your computer# because the tune method randomly shuffles the datatunedModelRMSE <‐ rmse(error) # 2.219642
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 10/19
This entry was posted in R and tagged Linear Regression, R, Support Vector Regression on October
23, 2014 [http://www.svm-tutorial.com/2014/10/support-vector-regression-r/] .
36 thoughts on “Support Vector Regression with R”
Good stu၀倀. How would this behave if for example, I wanted to predict some more X variables that
are not in the training set? Is this useful in those instances? - In that case, how?
Many thanks
You just need to use the predict method with two parameters: the trained model and your new data.
This will give you the predicted values. This is useful because that is our original goal, we want to
predict unseen data.
JoseNovember 8, 2014 at 12:35 pm
Alexandre KOWALCZYKNovember 8, 2014 at 12:53 pm
Post author
Joshua DunnJanuary 7, 2015 at 5:30 pm
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 11/19
I have tried predicting unseen data but it always seems to underestimate the e၀倀ect of it. For
example, with temperature as my x-variable, if my SVR has not seen temperatures below zero
degrees C (ie minus 2 degrees C) it e၀倀ectively predicts them as it would zero. Would you be able to
tell me what this is called or point me in a direction to solve this? Regards
For me it looks like you are over䁀䁀tting your model with your training data. What you should try is to
modify increase the weight of the regularization parameter (or use regularization if you were not)
Dear
Thank you very much. Actually I want to predict the future value of univariate time series by SVM. I
have used the library e1071. I am able to predict the value over the study period but i want to
forecast the future value.
Alexandre KOWALCZYKJanuary 8, 2015 at 9:42 pm
Post author
Md. Moyazzem HossainMarch 12, 2015 at 5:58 pm
What is the software on which you do the programming??? tks
In this case this is RStudio which can be downloaded here
linaFebruary 19, 2015 at 4:13 pm
Alexandre KOWALCZYKFebruary 19, 2015 at 10:56 pm
Post author
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 12/19
"we use the tune method to train models with ϵ=0,0.1,0.2,...,1 and cost = 22,23,24,...,29 which means
it will train 88 models (it can take a long time)"
Hello. Can you explain how the number 88 is calculated? Thank you.
There is 11 values of epsilon, and 8 values for the cost. We can associate each epsilon with the 8 cost
values to create 8 couples. As there is 11 epsilons, there is couples.
LizMarch 6, 2015 at 10:22 pm
Alexandre KOWALCZYKMarch 10, 2015 at 7:24 pm
Post author
11 × 8
Hello Mr. Kowalczyk.
This tutorial is very helpful. Actually i am trying to forecast the future value of a time-series data by
using SVR method, but i am quite confused how to perform it in R. Could you explain the steps on
how to do it?
Thank you
Thanks for your comment. Unfortunately I have never used SVR to forecast timeseries. However I
found this question and one of the answer is pointing to this article. As suggested in the answer you
will need to transform the classi䁀䁀cation problem to a regression one but this might be a good
Fakhrul AgustriwanApril 2, 2015 at 7:35 am
Alexandre KOWALCZYKApril 3, 2015 at 11:49 pm
Post author
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 13/19
starting point for you.
As I understand, SVM implemented in R uses the Radial Basis Kernel by default. Therefore, there is
another parameter (called gamma). How do you deal with this one? I think you should 䁀䁀t it also.
One article mentionned to take the median of pairwise distances between the learning points. (After
the scaling process)
You just need to add the gamma parameter in the tune function. There is an example in the e1071
package documentation :
obj < ‐ tune.svm(Species~., data = iris, gamma = 2^(‐1:1), cost = 2^(2:4))
loicApril 23, 2015 at 1:31 pm
Alexandre KOWALCZYKApril 23, 2015 at 9:33 pm
Post author
Ok thanks for your reply.
Using tune.svm I noticed that this function is very very long (around 3 seconds per con䁀䁀guration of
parameters for 1000 observations of 7 variables).
Surprisingly if you use svm(..., cross = 10) you can get the cross validation error for less than 0.5
second on the same data. So, I concluded that tune.svm was very badly coded, do you have any idea
about this issue?
Therefore I coded my own parameters tuning function using svm(...,cross=10).
Also, I have found several papers that use a BFGS optimization algorithm (on a log2 scale) instead of
grid search. I tried this, it turned out to be very e䁀䁀cient.
loicApril 24, 2015 at 12:06 pm
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 14/19
When you are using svm(..., cross = 10) you are performing a 10-fold cross-classi䁀䁀cation on the
training data. This is not the same as doing a grid search. If the method tune.svm is so slow, it is not
because it is poorly coded, but because it trains one svm model per combination of hyperparameter.
So if you want to try gamma=0.1,0.01 and C=1,10,100 for instance it will train 6 di၀倀erent svm models
([0.1,1][0.1,10][0.1,100] [0.01,1][0.01,10][0.01,100]) In other word, it will try each couple in the
cartesian product of the gamma set with the C set. If you try it for 10 values of gamma and 10 values
of C, it will train 100 models. Which should indeed be much slower than training only 10 models.
Alexandre KOWALCZYKApril 25, 2015 at 12:16 pm
Post author
That's not what I meant. I am aware of that of course. But actually, I made grid search "by hand" with
a loop on 10x10 values of gamma and C using svm(...,cross = 10). Therefore I called 100 times svm
and then keep the minimum cv error. The overall time it took was something like 10 times less than
calling once tune.svm() on a 10x10 grid.
That was what made me think this function was poorly coded or it might use so䁀䁀sticated techniques
I am not aware of.
I've been trying to 䁀䁀nd the reason in vain.
Actually, I am a bit doubtful about the results of svm(..., cross = 10), it seems that it does not
compute the sv error on a stochastic way and the results are only one decimal digit precise which is
weird comparing to tune.svm()
I can't really help you more without seeing your code. Maybe you can ask on stackover䁀䁀ow or cross
validated if you want to dig deeper and understand what happens in your particular case. Feel free
to post the link here afterward and I'll take a look.
loicApril 27, 2015 at 11:38 am
Alexandre KOWALCZYKApril 29, 2015 at 11:57 pm
Post author
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 15/19
Great tutorial for svm, clearly de䁀䁀ning its function as a classi䁀䁀er or a regressor, thanks Alexandre.
SpartanApril 29, 2015 at 8:06 pm
Thank you for this valuable post. If I have more than one X variable including some dummy variables
can I 䁀䁀t the SVR for that case?
Yes you can. SVR also works when X is multidimensional.
HAPJune 3, 2015 at 7:26 am
Alexandre KOWALCZYKJune 4, 2015 at 9:54 pm
Post author
Thanks. I'l try.
HAPJune 8, 2015 at 6:16 am
Hello,
I read your blog posts. I am not very clear about how to forecast future values of time series using
SVR. I looks to me that SVR 䁀䁀ts a model using training set. But how about using predict() to predict
future values ( n.ahead values ) in R? I couldnt 䁀䁀nd this feature so far..
IlgazJuly 3, 2015 at 2:17 pm
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 16/19
Sincerely, Ilgaz
I think you should take a look at the kernlab package as suggested in this stackexchange answer.
Alexandre KOWALCZYKJuly 3, 2015 at 11:22 pm
Post author
First of all, thanks for the very helpful tutorial. I'm using R 3.2.1, but svm doesn't work correctly. On
step 3, when I'm running this: model <- svm(Y ~ X , data), the error is :
Error in predict(ret, xhold, decision.values = TRUE) :
unused argument (decision.values = TRUE)
Can you please help me?
Thanks,
Hello. I don't have a lot of idea about this one. You might want to take a look at this answer and try
the provided solution. Otherwise I would advise you to try the code on another machine to see if it
works and if it does try to replicate the environment on your machine. Best regards.
AseelJuly 6, 2015 at 8:54 pm
Alexandre KOWALCZYKJuly 6, 2015 at 11:33 pm
Post author
AseelJuly 7, 2015 at 1:25 am
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 17/19
Pingback: Support Vector Regression in R | logicalerrors
I really appreciate your replay. I think the problem is there is same function name in two packages,
for example predict() function in both ".GlobalEnv" and "package:stats" packages.
I will try to 䁀䁀gure out how to solve that.
Thanks a lot,
I've found that I have function with the same name with predict. So, simply, I've copied my function
to another name and remove predict function. That was making the confusion.
Thanks again for helping me.
AseelJuly 7, 2015 at 1:38 am
Hi Alexandre. Thank you so much for all the information, I have a few questions.
1. Can I use any kínd of variables in a SVM ? Continuous, categorical?
2. If I am using a SVM to classify two groups, is there a way to get a probability of assignment to each
group?
3. How do you validate that the SVM is a good model?
1. Yes. For continuous data it is called SVR and SVM for categorical data
2. Yes. Most framework provide a method "predict probabilities" to do so
3. You use a score to measure the quality of your model, if you want to learn more I recommend you
this book.
EspartacoJuly 17, 2015 at 3:34 am
Alexandre KOWALCZYKJuly 19, 2015 at 2:36 pm
Post author
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 18/19
Hi Alexandre,
Thanks for such a comprehensive tutorial. Much appreciated. I am trying to SVR for predicting time
series. As mentioned in your post, tune() shu䁀၀es the data. Is there any option or way to not to
shu䁀၀e the data?
Hello. You can specify a tunecontrol parameter to specify the behavior of the tune method. I think
tune.control(sampling = "䁀䁀x") might suit your need.
Yes. Worked perfectly!
DannyAugust 12, 2015 at 1:43 am
Alexandre KOWALCZYKAugust 14, 2015 at 10:31 pm
Post author
DannyAugust 15, 2015 at 1:23 am
hi Alexandre I am asking for using ls-svm in regression are there any R packages support it
tahirAugust 14, 2015 at 3:40 pm
Alexandre KOWALCZYKAugust 14, 2015 at 10:24 pm
Post author
19/08/2015 Support Vector Regression with R SVM Tutorial
http://www.svmtutorial.com/2014/10/supportvectorregressionr/ 19/19
Hello, you can use the function lssvm available in the kernlab package.
thank you very much, but when I use lssvm I get this message
(Using automatic sigma estimation (sigest) for RBF or laplace kernel
Error in if (n != dim(y)[1]) stop("Labels y and data x dont match") :
argument is of length zero)
tahirAugust 18, 2015 at 6:32 am