cs 6140: machine learning - college of computer … · what we learned last @me • evaluaon...
TRANSCRIPT
![Page 1: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/1.jpg)
CS6140:MachineLearningSpring2017
Instructor:LuWangCollegeofComputerandInforma@onScience
NortheasternUniversityWebpage:www.ccs.neu.edu/home/luwang
Email:[email protected]
![Page 2: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/2.jpg)
Logis@cs
• Assignment1isout– Due2/9/2017– Startearly!
![Page 3: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/3.jpg)
Whatwelearnedlast@me
• Evalua@onmetrics
• DecisionTree
• Genera@veModels
• Genera@veModelandDiscrimina@veModel
• Logis@cRegression
![Page 4: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/4.jpg)
![Page 5: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/5.jpg)
![Page 6: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/6.jpg)
![Page 7: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/7.jpg)
ROCPlot
• Sensi@vity=a/(a+b)=Recall– Trueposi@verate
• 1-Specificity=1-d/(c+d)=c/(c+d)– Falseposi@verate
![Page 8: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/8.jpg)
![Page 9: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/9.jpg)
DecisionTree
• Playtennis?
• Eachinternalnode:testonefeatureXi• Eachbranchfromanode:selectsonevalueforXi• Eachleafnode:predictY(orP(Y|X∈leaf))
![Page 10: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/10.jpg)
Top-DownInduc@onofDecisionTrees
• Whicha_ributetouseforsplit?
![Page 11: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/11.jpg)
Top-DownInduc@onofDecisionTrees
• Whicha_ributetouseforsplit?• Goodsplitifwearemorecertainaboutclassifica@ona`ersplit– Determinis@cgood(alltrueorallfalse)– Uniformdistribu@onbad– Whataboutdistribu@oninbetween?
![Page 12: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/12.jpg)
Informa@onGain
• Gain(S,A)=expectedreduc@oninentropyduetosor@ngonA
![Page 13: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/13.jpg)
Informa@onGain
![Page 14: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/14.jpg)
Overficng
![Page 15: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/15.jpg)
![Page 16: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/16.jpg)
Bayesianmodel
• H:Hypothesisspaceofpossibleconcepts• X:nexamplesofaconceptC• EvaluatehypothesesgivendatausingBayes’rule:
![Page 17: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/17.jpg)
NaïveBayes
![Page 18: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/18.jpg)
MaximumLikelihoodEs@ma@on
![Page 19: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/19.jpg)
MaximumLikelihoodEs@ma@on
![Page 20: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/20.jpg)
[email protected]@veModel
• P(Y|X)=p(X,Y)/P(X)
• Genera@vemodel– LearnP(X,Y)fromtrainingsample– P(X,Y)=P(Y)P(X|Y)– Specifieshowtogeneratetheobservedfeaturesxfory
• Discrimina@vemodel– LearnP(Y|X)fromtrainingsample– Directlymodelsthemappingfromfeaturesxtoy
![Page 21: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/21.jpg)
Logis@cRegression
![Page 22: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/22.jpg)
Sigmoidfunc@on
• Defini@on
![Page 23: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/23.jpg)
Logis@cRegression
![Page 24: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/24.jpg)
Logis@cRegression
![Page 25: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/25.jpg)
MaximizingLogLikelihood
![Page 26: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/26.jpg)
![Page 27: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/27.jpg)
![Page 28: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/28.jpg)
![Page 29: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/29.jpg)
GradientDescent• Example
![Page 30: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/30.jpg)
ChangingStepSize
![Page 31: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/31.jpg)
AddingPrior
![Page 32: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/32.jpg)
![Page 33: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/33.jpg)
Today’sOutline
• Perceptron(andkernels)
• SupportVectorMachines
![Page 34: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/34.jpg)
Perceptron
[SomeoftheslidesareborrowedfromAlexSmola’stutorial]
![Page 35: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/35.jpg)
BiologyandLearning
• Idea1:Goodbehaviorshouldberewarded,badbehaviorpunished(ornotawarded).– Raisingadog.
• Idea2:Correlatedeventsshouldbecombined.– Babieslearnlanguage.
![Page 36: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/36.jpg)
BiologyandLearning
• Idea1:Goodbehaviorshouldberewarded,badbehaviorpunished(ornotawarded).– Raisingadog.
• Idea2:Correlatedeventsshouldbecombined.– Babieslearnlanguage.
• TrainingMechanisms– Behavioralmodifica@onofindividuals(learning)
• Feedingthedog,thenthedoglearnstostandandsit.– Hard-codedbehaviorinthegenes(ins@nct)
• Thewronglycodedanimaldies.
![Page 37: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/37.jpg)
Neurons
![Page 38: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/38.jpg)
Perceptron
![Page 39: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/39.jpg)
Perceptron
• Weightedcombina@on– Theoutputoftheneuronisalinearcombina@onoftheinputs
• DecisionFunc@on– Attheendtheresultsarecombinedinto
![Page 40: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/40.jpg)
Perceptron
• Anabstractmodelistoassumethat
– Wherewistheweight,xisthefeaturevectors
– bisthebias,• BiologicalInterpreta@on– Theweightswicorrespondtothesynap@cweights,themul@plica@oncorrespondstotheprocessingofinputsviathesynapses,andthesumma@onisthecombina@onofsignalsinthecellbody(soma).
![Page 41: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/41.jpg)
LearningGoal:LinearSepara@on
![Page 42: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/42.jpg)
PerceptronAlgorithm
![Page 43: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/43.jpg)
PerceptronAlgorithm• Nothinghappensifweclassify(xi,yi)correctly• Ifweseeincorrectlyclassifiedobserva@onweupdatew
andb• Posi@vereinforcementofobserva@ons
![Page 44: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/44.jpg)
PerceptronAlgorithm• Aboutthesolu@on– Weightvectorislinearcombina@onofobserva@onsxi:
![Page 45: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/45.jpg)
PerceptronAlgorithm• Aboutthesolu@on– Classifica@oncanbewri_enintermsofdotproducts:
![Page 46: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/46.jpg)
![Page 47: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/47.jpg)
![Page 48: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/48.jpg)
Pseudocode
![Page 49: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/49.jpg)
TheXORProblem
![Page 50: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/50.jpg)
TheXORProblem
• Perceptronscannotlearnsuchlinearlyinseparablefunc@ons!
![Page 51: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/51.jpg)
Problem
• Linearfunc@onsareo`entoosimpletoprovidegoodes@mators.
![Page 52: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/52.jpg)
Problem
• Linearfunc@onsareo`entoosimpletoprovidegoodes@mators.
• Idea:– Maptoahigherdimensionalfeaturespacevia
– Replaceevery by intheperceptronalgorithm.
![Page 53: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/53.jpg)
![Page 54: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/54.jpg)
![Page 55: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/55.jpg)
PerceptrononFeatures
![Page 56: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/56.jpg)
ProblemswithConstruc@ngFeatures
![Page 57: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/57.jpg)
ProblemswithConstruc@ngFeatures
• Needtobeanexpertinthedomain(e.g.Chinesecharacters).
• Canbeexpensivetocompute.
![Page 58: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/58.jpg)
PolynomialFeatures• Dimension=1• Dimension=2
• Dimension=d(skipproof)
![Page 59: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/59.jpg)
Kernels
• Defini@on• Akernelfunc@on isasymmetricfunc@oninitsargumentsforwhichthefollowingpropertyholds
![Page 60: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/60.jpg)
Somechoicesofkernelfunc@ons
RBFkernel:Radialbasisfunc@onkernel
![Page 61: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/61.jpg)
LinearKernel
![Page 62: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/62.jpg)
LaplacianKernel
![Page 63: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/63.jpg)
GaussianKernel
![Page 64: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/64.jpg)
KernelPerceptron
![Page 65: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/65.jpg)
LinearSeparators
• Whichoftheselinearseparatorsisop@mal?
[SomeoftheslidesareborrowedfromDavidSontag’slecture]
![Page 66: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/66.jpg)
![Page 67: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/67.jpg)
Outline
• Perceptron(andkernels)
• SupportVectorMachines
![Page 68: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/68.jpg)
SupportVectorMachine(SVM)
• SVMs(Vapnik,1990’s)choosethelinearseparatorwiththelargestmargin.
Hyperplane
![Page 69: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/69.jpg)
SupportVectorMachine(SVM)
• Reasons:
– Intui@on– Theore@calguarantee(skiphere)– Inprac@caltasks:SVMbecamefamouswhen,usingimagesasinput,it
gaveaccuracycomparabletoneural-networkwithhand-designedfeaturesinahandwri@ngrecogni@ontask.
![Page 70: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/70.jpg)
SupportVectorMachine(SVM)
• Howtofindthehyperplane?
Hyperplane
![Page 71: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/71.jpg)
Planes
• Aplanecanbespecifiedasthesetofpointsgivenby
![Page 72: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/72.jpg)
Planes
• Aplanecanbespecifiedasthesetofpointsgivenby
NormalVector:decidethedirec@onoftheplane
![Page 73: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/73.jpg)
Normaltoaplane
Lengthofthevector
![Page 74: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/74.jpg)
Scaleinvariance
![Page 75: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/75.jpg)
![Page 76: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/76.jpg)
Whatisthedistance ?γ
Finalresult:canmaximizemarginbyminimizing
![Page 77: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/77.jpg)
Supportvectormachines
![Page 78: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/78.jpg)
Whatifthedataisnotlinearlyseparable?
![Page 79: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/79.jpg)
Whatifthedataisnotlinearlyseparable?
• Morefeatures
![Page 80: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/80.jpg)
Whatifthedataisnotlinearlyseparable?
Oldobjec@ve
Newobjec@ve
![Page 81: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/81.jpg)
Whatifthedataisnotlinearlyseparable?
Oldobjec@ve
Newobjec@ve
Jointlyminimizew.wandnumberoftrainingmistakes!
![Page 82: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/82.jpg)
Whatifthedataisnotlinearlyseparable?
![Page 83: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/83.jpg)
Allowingforslack:“So`marginSVM”
![Page 84: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/84.jpg)
Allowingforslack:“So`marginSVM”
![Page 85: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/85.jpg)
PopularToolsforSVMs
• LIBSVM(c++)– h_ps://www.csie.ntu.edu.tw/~cjlin/libsvm/
• SVMlight(c)– h_p://svmlight.joachims.org/
• Scikit-learn(python)– h_p://scikit-learn.org/
![Page 86: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/86.jpg)
PopularToolsforSVMs
• Torch(LuaJIT)– h_p://torch.ch/
• Spider(Matlab)
• Weka(Java)
![Page 87: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/87.jpg)
Howdoweop@mizetheobjec@ve?
• Quadra@cprogramming
![Page 88: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/88.jpg)
Kernels
• Defini@on• Akernelfunc@on isasymmetricfunc@oninitsargumentsforwhichthefollowingpropertyholds
![Page 89: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/89.jpg)
Howdoweop@mizetheobjec@ve?
• Quadra@cprogramming
Noplacetoapplythekerneltrick
![Page 90: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/90.jpg)
Constrainedop@miza@on
![Page 91: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/91.jpg)
Constrainedop@miza@on
![Page 92: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/92.jpg)
Constrainedop@miza@on
![Page 93: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/93.jpg)
Lagrangemul@pliers–Dualvariables
• LagrangeMul@pliers
![Page 94: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/94.jpg)
Howdowesolvewithconstraints?
• LagrangeMul@pliers
![Page 95: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/95.jpg)
Lagrangemul@pliers–Dualvariables
![Page 96: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/96.jpg)
BacktoSVM(hardmargin)
![Page 97: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/97.jpg)
DualSVMderiva@on
![Page 98: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/98.jpg)
DualSVMderiva@on
Slater’scondi@onfromconvexop@miza@onguaranteesthatthesetwoop@miza@onproblemsareequivalent!(skipproof)
![Page 99: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/99.jpg)
DualSVMderiva@on
![Page 100: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/100.jpg)
DualSVMderiva@on
![Page 101: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/101.jpg)
Togetwandb
![Page 102: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/102.jpg)
Classifica@onruleusingdualsolu@onUsingdualsolu@ondotproduct
![Page 103: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/103.jpg)
Dualforthenon-separablecase
![Page 104: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/104.jpg)
Dualforthenon-separablecase
![Page 105: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/105.jpg)
Howtointerpretdualform
![Page 106: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/106.jpg)
Backtotheques@on:Whatifthedataisnotlinearlyseparable?
![Page 107: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/107.jpg)
Forexample:Higherorderpolynomials
![Page 108: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/108.jpg)
Dualformula@ononlydependsondot-productsofthefeatures!
![Page 109: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/109.jpg)
Dualformula@ononlydependsondot-productsofthefeatures!
![Page 110: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/110.jpg)
Kernels
• Defini@on• Akernelfunc@on isasymmetricfunc@oninitsargumentsforwhichthefollowingpropertyholds
![Page 111: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/111.jpg)
KernelTrick
![Page 112: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/112.jpg)
So`marginSVMwithkernel
![Page 113: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/113.jpg)
CommonkernelsforSVM
AkaGaussianRadialbasisfunc@on(RBF)kernel
![Page 114: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/114.jpg)
Overficng
• Hugefeaturespacewithkernels:shouldweworryaboutoverficng?– SVMobjec@veseeksasolu@onwithlargemargin– Goodtheore@calguarantee– Buteverythingoverfitssome@mes
![Page 115: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/115.jpg)
Overficng
• Hugefeaturespacewithkernels:shouldweworryaboutoverficng?– SVMobjec@veseeksasolu@onwithlargemargin– Goodtheore@calguarantee– Buteverythingoverfitssome@mes
• Cancontrolby:– SecngC– Choosingabe_erkernel– Varyingparametersofthekernels
![Page 116: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/116.jpg)
Dualforthenon-separablecase
![Page 117: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/117.jpg)
LinearSVMC=50
![Page 118: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/118.jpg)
LinearSVMC=50
![Page 119: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/119.jpg)
LinearSVMC=50
![Page 120: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/120.jpg)
LinearSVMC=50
![Page 121: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/121.jpg)
Insights
• ChangingC– ForcleandataCdoesn’tma_ermuch.– Fornoisydata,largeCleadstonarrowmargin(SVMtriestodoagoodjobatsepara@ng,eventhoughitisn’tpossible)
![Page 122: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/122.jpg)
Insights
• ChangingC– ForcleandataCdoesn’tma_ermuch.– Fornoisydata,largeCleadstonarrowmargin(SVMtriestodoagoodjobatsepara@ng,eventhoughitisn’tpossible)
• Noisydata– Cleandatahasfewsupportvectors– Noisydataleadstodatainthemargins– Moresupportvectorsfornoisydata
![Page 123: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/123.jpg)
GaussianRBFKernelwithC=0.1
![Page 124: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/124.jpg)
GaussianRBFKernelwithC=0.2
![Page 125: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/125.jpg)
GaussianRBFKernelwithC=0.4
![Page 126: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/126.jpg)
GaussianRBFKernelwithC=0.8
![Page 127: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/127.jpg)
GaussianRBFKernelwithC=1.6
![Page 128: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/128.jpg)
GaussianRBFKernelwithC=3.2
![Page 129: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/129.jpg)
GaussianRBFKernelwithC=6.4
![Page 130: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/130.jpg)
GaussianRBFKernelwithC=12.8
![Page 131: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/131.jpg)
Insights
• ChangingC– ForcleandataCdoesn’tma_ermuch.– Fornoisydata,largeCleadstomorecomplicatedmargin(SVMtriestodoagoodjobatsepara@ng,eventhoughitisn’tpossible)
– OverfiMngforlargeC• Noisydata– Cleandatahasfewsupportvectors– Noisydataleadstodatainthemargins– Moresupportvectorsfornoisydata
![Page 132: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/132.jpg)
CommonkernelsforSVM
AkaGaussianRadialbasisfunc@on(RBF)kernel
![Page 133: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/133.jpg)
GaussianRBFwithdifferent
![Page 134: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/134.jpg)
GaussianRBFwithdifferent
![Page 135: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/135.jpg)
GaussianRBFwithdifferent
![Page 136: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/136.jpg)
GaussianRBFwithdifferent
![Page 137: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/137.jpg)
Insights
• Changing– Forcleandata, doesn’tma_ermuch.– Fornoisydata,small leadstomorecomplicatedmargin(SVMtriestodoagoodjobatsepara@ng,eventhoughitisn’tpossible)
– Lotsofoverficngforsmall• Noisydata– Cleandatahasfewsupportvectors– Noisydataleadstodatainthemargins– Moresupportvectorsfornoisydata
![Page 138: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/138.jpg)
Homework(partofassignment2)
• Studythe“Sequen@alMinimalOp@miza@on”algorithmandimplementanSVMclassifierbyyourself
• References– h_p://cs229.stanford.edu/materials/smo.pdf– FastTrainingofSupportVectorMachinesusingSequen@alMinimalOp@miza@on
– h_p://research.microso`.com/pubs/68391/smo-book.pdf
![Page 139: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/139.jpg)
Whatwelearnedtoday
• Perceptron(andkernels)
• SupportVectorMachines
![Page 140: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model](https://reader031.vdocument.in/reader031/viewer/2022021900/5b5e39c77f8b9a8b4a8bef80/html5/thumbnails/140.jpg)
Homework
• ReadMurphyCH14.1-14.2,14.4-14.5.• Assignment1isout.Dueintwoweeks.