support vector machines joseph gonzalez texpoint fonts used in emf. read the texpoint manual before...
Post on 19-Dec-2015
221 views
TRANSCRIPT
![Page 1: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/1.jpg)
Support Vector Machines
Joseph Gonzalez
![Page 2: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/2.jpg)
From a linear classifier to ...
*One of the most famous slides you will see, ever!
![Page 3: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/3.jpg)
The Big Idea
O
X
O
O
X
X
X
X
X
X
O
O
OO
O
O
![Page 4: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/4.jpg)
Maximum margin
Maximum possible separation between positive and negative training examples
*One of the most famous slides you will see, ever!
![Page 5: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/5.jpg)
Geometric Intuition
O
X
O
OO
XX
X
SUPPORT VECTORS
![Page 6: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/6.jpg)
Geometric Intuition
O
X
X
O
OO
XX
X
SUPPORT VECTORS
![Page 7: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/7.jpg)
Geometric Intuition
O
X
XO
O
O
XX
X
SUPPORT VECTORS
![Page 8: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/8.jpg)
Primal Versionmin ||w||
2 +C ∑ξs.t. (w.x + b)y ≥ 1-ξ
ξ ≥ 0
![Page 9: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/9.jpg)
DUAL Version
Where did this come from?Remember Lagrange Multipliers
Let us “incorporate” constraints into objectiveThen solve the problem in the “dual” space of lagrange multipliers
max ∑α -1/2 ∑αiαjyiyjxixj
s.t. ∑αiyi = 0C ≥ αi ≥ 0
![Page 10: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/10.jpg)
Primal vs Dual
Number of parameters?large # features?large # examples?
for large # features, DUAL preferredmany αi can go to zero!
max ∑α -1/2 ∑αiαjyiyjxixj
s.t. ∑αiyi = 0C ≥ αi ≥ 0
min ||w||2 +C ∑ξ
s.t. (w.x + b)y ≥ 1-ξξ ≥ 0
![Page 11: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/11.jpg)
DUAL: the “Support vector” version
How do we find α?
Quadratic programming
How do we find C?
Cross-validation!
Wait... how do we predict y for a new point x??
How do we find w?
How do we find b?
y = sign(w.x+b)
w = Σi αi yi xi
max ∑α - 1/2 ∑αiαjyiyjxixj
s.t. ∑αiyi = 0C ≥ αi ≥ 0
y = sign(Σi αi yi xi xj + b)
![Page 12: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/12.jpg)
max α1 + α2 + 2α1α2 - α12/2 - 4α22
s.t. α1-α2 = 0C ≥ αi ≥ 0
“Support Vector”s?
O
X
α1
α2
max ∑α - 1/2 ∑αiαjyiyjxixj
s.t. ∑αiyi = 0C ≥ αi ≥ 0
(0,1)
(2,2)max ∑α - α1α2(-1)(0+2)- 1/2 α12(1)(0+1) - 1/2 α22(1)(4+4)
w = Σi αi yi xi
w = .4([0 1]-[2 2]) =.4[-2 -1 ]
y=w.x+bb = y-w.xx1: b = 1-
.4 [-2 -1][0 1] = 1+.4 =1.4
b
4/5
α1=α2=αmax 2α -5/2α2
max 5/2α(4/5-α) 0 2/5
α1=α2=2/5
![Page 13: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/13.jpg)
“Support Vector”s?
O
X
α1
α2
max ∑α - 1/2 ∑αiαjyiyjxixj
s.t. ∑αiyi = 0C ≥ αi ≥ 0
(0,1)
(2,2)
Oα3
What is α3? Try this at home
![Page 14: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/14.jpg)
Playing With SVMS
• http://www.csie.ntu.edu.tw/~cjlin/libsvm/
![Page 15: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/15.jpg)
More on Kernels
• Kernels represent inner products– K(a,b) = a.b– K(a,b) = φ(a) . φ(b)
• Kernel trick is allows extremely complex φ( ) while keeping K(a,b) simple
• Goal: Avoid having to directly construct φ( ) at any point in the algorithm
![Page 16: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/16.jpg)
Kernels
Complexity of the optimization problem remains only dependent on the dimensionality of the input space and not of the feature space!
![Page 17: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/17.jpg)
Can we used Kernels to Measure Distances?
• Can we measure distance between φ(a) and φ(b) using K(a,b)?
![Page 18: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/18.jpg)
Continued:
![Page 19: Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A AA A AA](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d2e5503460f94a05edd/html5/thumbnails/19.jpg)
Popular Kernel Methods
• Gaussian Processes• Kernel Regression (Smoothing)
– Nadarayan-Watson Kernel Regression