CS480/680: Intro to MLLecture 07: Hard-margin SVM

2020-06-02 Yao-Liang Yu1

Outl ine

• Maximum Margin

• Lagrangian Dual

• Alternative View

2020-06-02 Yao-Liang Yu2

Perceptron revisited

2020-06-02 Yao-Liang Yu3

• Two classes: y = 1 or y = -1

• Assuming linearly separable• exist w and b such that for all i,

yi(wTxi + b) > 0

• Find any such w and b feasibility problemmin

w,b0

s.t. 8i, yi(w>xi + b) > 0

Margin

• Take any linear separating hyperplane H

for all i, yi(wTxi + b) > 0

• Move H until it touches some positive point, H1

increase b say

• Move H until it touches some negative point, H-1

decrease b say2020-06-02 Yao-Liang Yu4

margin = dist(H1,H) ∧ dist(H-1,H)

minx,z

kx� zk2 ���� w>

kwk2(x� z)

���2=

|t|kwk2

s.t. w>x+ b = t

w>z+ b = 0

Put into formula

H : wTx + b = 0H1 : wTx + b = tH-1 : wTx + b = -s

What is the distance betweenH1 and H ?

2020-06-02 Yao-Liang Yu5

equality is attainedx = w

kwk22(t� b)

z = wkwk2

2(�b)

Put into formula

H : wTx + b = 0H1 : wTx + b = tH-1 : wTx + b = -t

What is the distance betweenH1 and H ?

2020-06-02 Yao-Liang Yu6

Put into formula

H : wTx + b = 0H1 : wTx + b = 1H-1 : wTx + b = -1

What is the distance betweenH1 and H ?

2020-06-02 Yao-Liang Yu7

Maximum Margin

2020-06-02 Yao-Liang Yu8

maxw,b

1

kwk2s.t. 8i, yi(w>xi + b) � 1

Important facts. • For any f,

• For positive f,

• For s. monotone g, minw

f(w) ⌘ minw

g(f(w))

maxw

1

f(w)=

1

minw f(w)

maxw

f(w) = �minw

�f(w)

Hard-margin Support Vector Machines

2020-06-02 Yao-Liang Yu9

maxw,b

1

kwk2s.t. 8i, yi(w>xi + b) � 1

margin 1

kwk2

Quadratic programming

minw,b

1

2kwk22

s.t. 8i, yi(w>xi + b) � 1

Comparing to perceptron

2020-06-02 Yao-Liang Yu10

minw,b

0

s.t. 8i, yi(w>xi + b) > 0

Linear programming

Support Vectors

• Those touch the parallel hyperplanes H1and H-1

• Usually only a handful

• Entirely determine the hyperplanes!

2020-06-02 Yao-Liang Yu11

Existence and uniqueness

2020-06-02 Yao-Liang Yu12

minw,b

1

2kwk22

s.t. 8i, yi(w>xi + b) � 1

• Always exists a minimizer w and b (if linearly separable)

• The minimizer w is unique (strict convexity of )

• The minimizer b is also unique (why?)

12kwk22

Outl ine

• Maximum Margin

• Lagrangian Dual

• Alternative View

2020-06-02 Yao-Liang Yu13

Lagrangian

2020-06-02 Yao-Liang Yu14

minw,b

max↵�0

1

2kwk22 �

X

i

↵i[yi(w>xi + b)� 1]

Lagrangian

Lagrangian multiplier[dual variable][primal variable]

Primalminw,b

1

2kwk22

s.t. 8i, yi(w>xi + b) � 1

Deriving the dual

2020-06-02 Yao-Liang Yu15

minw,b

max↵�0

1

2kwk22 �

X

i

↵i[yi(w>xi + b)� 1]

max↵�0

minw,b

1

2kwk22 �

X

i

↵i[yi(w>xi + b)� 1]

@

@b=

X

i

↵iyi = 0@

@w= w �

X

i

↵iyixi = 0

The dual problem

2020-06-02 Yao-Liang Yu16

max↵�0

X

i

↵i �1

2

���X

i

↵iyixi

���2

2

s.t.X

i

↵iyi = 0 Only need dot product in the dual !

min↵�0

1

2

X

i

X

j

↵i↵jyiyjx>i xj �

X

k

↵k

s.t.X

i

↵iyi = 0Dual

Rn

Support Vectors

2020-06-02 Yao-Liang Yu17

w =X

i

↵iyixi

↵i > 0

Outl ine

• Maximum Margin

• Dual

• Alternative View

2020-06-02 Yao-Liang Yu18

An dual view

2020-06-02 Yao-Liang Yu19

Convex sets and Convex hul lConvex set. A point set is convex if the line segment [x,y] connecting any two points x and y in C lies entirely in C.

Convex hull. Smallest convex set containing C.

2020-06-02 Yao-Liang Yu20

C 2 Rd

ch(C) :=

(X

i

↵ixi : xi 2 C,↵i � 0,X

i

↵i = 1

).

Separating scale from direct ion

2020-06-02 Yao-Liang Yu21

min↵�0

1

2

���X

i

↵iyixi

���2

2�X

i

↵i

s.t.X

i

↵iyi = 0

minr�0

min↵̄22�

r2

2

���X

i

↵̄iyixi

���2

2� 2r

s.t.X

i

↵̄iyi = 0

<latexit sha1_base64="LJVLRl4GmE/GZSzWmxkW6CvWh+k=">AAACw3icdVHLbtQwFHXCqwyvARYs2FiMQGyIkqgINpWq8lwWiWkrjdPI8dzMWGM7wXagI2P+ic/hW9jgZEYCWriSpXPPffrcqhXc2DT9EcWXLl+5em3n+ujGzVu374zv3jsyTacZTFkjGn1SUQOCK5habgWctBqorAQcV6tXffz4M2jDG/XRrlsoJF0oXnNGbaDK8XciuSqdxmQBn3Dq8Te8YUhFNaGiXVJMuMJEw9zlnrwGYa nHT0JarSlz+jT3gcfkgC/IV2I6WXL8uzY4656Q1C6r2p353hlSy/w0x89wHiaT0RDX0pnEJpvm/2m0h9NyPEmTdDB8EWRbMEFbOyzHP8m8YZ0EZZmgxsyytLWFo9pyJsCPSGegpWxFFzALUFEJpnCDtB4/Dswc140OT1k8sH9WOCqNWcsqZPZ/MOdjPfmv2Kyz9cvCcdV2FhTbDKo7gW2D+zvhOdfArFgHQJnmYVfMljRIbsM1R0TBF9ZISdXc9afxs6xwxMKZHTZ0PeUmmfdBrey8NhfBUZ5ku8nzD7uT/YOtbjvoIXqEnqIMvUD76D06RFPEogfRXvQ2ehe/iVexju0mNY62NffRXxb7X0ub2wU=</latexit>

↵ = r↵̄

<latexit sha1_base64="LTS5UXJY78P1nzhDFYNz4RAXGjI=">AAACJ3icbVBNSxxBFOxR48cadU3IyUvjEshpmZGVeBGW5JKjgawKO8Pypuet29jdM3S/SbI082NyVH9MbmKO+R25pPfjkKgFDdVV7/GKyislHcXxr2hlde3F+sbmVmv75c7uXnv/1bkraytwIEpV2sscHCppcECSFF5WFkHnCi/y648z/+IrWidL84WmFWYarowcSwEUpFH7TQqqmgA/5ZanOdjFd9TuxN14Dv6UJEvSYUucjdp/0qIUtUZDQoFzwySuKPNgSQqFTSutHVYgruEKh4Ea0OgyP4/f8LdBKfi4tOEZ4nP13w0P2rmpzsOkBpq4x95MfM4b1jQ+ybw0VU1oxOLQuFacSj7rghfSoiA1DQSElSErFxOwICg01koNfhOl1mAKn1osmmGS+ZTwO80T+pnkO0nThLaSx908JedH3aTXPf7c6/Q/LHvbZAfskL1jCXvP+uwTO2MDJphnP9gtu4tuop/RffSwGF2Jljuv2X+Ifv8F+mCmkw==</latexit>

r =2

kP

i ↵̄iyixik22

<latexit sha1_base64="DjWuACCqz2igWpudl5dZEnLh7ks=">AAACTHicbVBNa9wwFJS3bZps+rFNj7mILIWcFntJaSkUQnLpMYVuEli55ll+zopIspHkJIvi/5Jf02Pba/9Hb6EQebOHfHRAMJp5D40mr6WwLo7/RL0nT5+tPF9d66+/ePnq9eDNxqGtGsNxwitZmeMcLEqhceKEk3hcGwSVSzzKT/c7/+gMjRWV/ubmNaYKTrQoBQcXpGzwydDPlJUGuB+3nl0y26hMUJaDYSDrGYTLvBMUuFle+os2E+wyG38ft9lgGI/iBehjkizJkCxxkA2uWVHxRqF2XIK10ySuXerBOMEltn3WWKyBn8IJTgPVoNCmfvHHlr4LSkHLyoSjHV2odzc8KGvnKg+TXVT70OvE/3nTxpUfUy903TjU/PahspHUVbQrjBbCIHdyHghwI0JWymcQCnOh1j7TeM4rpUAXnhks2mmSeubwwi0S+k7yw6Tt2koedvOYHI5Hyc7o/ded4e7esrdVskm2yDZJyAeyS76QAzIhnFyRH+QX+R39jP5G19G/29FetNx5S+6ht3IDfDy1Vw==</latexit>

Scale r does not matter

2020-06-02 Yao-Liang Yu22

min↵̄22�

r2

2

���X

i

↵̄iyixi

���2

2� 2r

s.t.X

i

↵̄iyi = 0

<latexit sha1_base64="LkUqqtCsLjlr23w0pCeSwG2dCKQ=">AAADVXicdVJNb9QwEE12Synho1s4csAiLeLCKom2gkulqoDECYpEP6Q4jRzH2Vh1nMh26K6MOfGn+CWIH4OEk13BbgsjWR6/eTNjz3PWMCpVEPx0B8ONW5u3t+54d+/df7A92nl4KutWYHKCa1aL8wxJwignJ4oqRs4bQVCVMXKWXb7u4mefiZC05p/UvCFJhaacFhQjZaF09B1WlKcaZkhAxJoSAUg50JGBbwhTyIBnXwEsBMIalhhxTFgsSJ5ocREZY2kAHtEp/AJlW6UU/C1jD/MOqJAqs0LPTHfoqWl0EYH1Yi9AJGwl6PV0UWk5VuNF6//UPQBBOvKDcdAbuOmES8d3lnac7rjfYF7jtiJcYYakjMOgUYlGQlHMiPFgK0mD8CWakti6HFVEJrqfsQF7FslBUQu7uAI9upqhUSXlvMoss3uEvB7rwH/F4lYVrxJNedMqwvGiUdEyoGrQCQZyKghWbG4dhAW1dwW4RFYRZWX1ICdXuK4qxHMN7TBNHCYaKjJT/Q11B2k/NGuv04vhm9XsP4pESZwxy7NlJFFZPQsOYGk3vetHu2bPg4KhxvZCVJIOHk/2YakCs9rVNrSMlhENr3IbChtlrPn2v3hWt/C6Sjed02gcTsb7Hyf+4dFSwS3nsfPUee6Ezkvn0HnnHDsnDnafuG/d9+6HwY/Br+HGcHNBHbjLnEfOmg23fwP2VRKR</latexit>

P := {i : yi = 1}N := {i : yi = �1}

minµ2�,⌫2�

1

2

���X

i2P

µixi �X

j2N

⌫jxj

���2

2

<latexit sha1_base64="dShlqjHrWB65msjpIEXzRJ4WpgY=">AAADQnicbVFdb9MwFHXC1yiDdfDIi0U2xANUSdQJJECaBg88oSLRbVIcIsd1Wne2E8UOa2XME7+FX8Kf4C/whnjlAaetpnbjSpavzz3Xx74nrzhTOgx/ev616zdu3tq63bmzfffeTnf3/rEqm5rQISl5WZ/mWFHOJB1qpjk9rWqKRc7pSX72pq2ffKa1YqX8qOcVTQUeS1YwgrWDsu53JJjMDBINYhK9pVzjpxDJBl4cLXz8FaIJwZJQntR0lBpU1JiYyJrYWoiO2Bh9QaoRmWGuCw4cJpqMIYH1JC/MzGYMPoNLxrRlvLetRDZdY0xX92TxJ7Mp1mp0sm4Q9sJFwKtJtEoCsIpBtut9Q6OSNIJKTThWKonCSqcG15oRTm0HNYpWmJzhMU1cKrGgKjWLgVq475ARLMraLanhAl3vMFgoNRe5Y7ZfUJdrLfi/WtLo4kVqmKwaTSVZChUNh7qErTtwxGpKNJ+7BJOaubdCMsFu2tp52EGSnpNSCCxHBrnh2CRyZmg604sXmhYyQWQ3fmeWw7Tr3RcTjtMk547nrlFU5+UsfI0mbjN7Qbxn9zuo5rhyWpgp2sK9/gGa6NCuqzpBx2g4Neh85EpRpa2LIF76Fl126WpyHPeifu/gQz84PFo5uAUegkfgCYjAc3AI3oEBGALibXux99J75f/wf/m//T9Lqu+teh6AjfD//gPKPA/h</latexit>

conv({xi : i 2 P})

<latexit sha1_base64="FJmPAEi1Bm1ojfS/T3CbyJS9mFE=">AAACx3icbVFNaxsxEJW3X6n75bTHXkQ3gZSC2TUOLYVCaC/tzYU6CawWo9WOYxGttEhax0boUOgf6K/ptf0Z/TfVrn1wkj4Qerw3o5nRFLXgxibJ31505+69+w/2HvYfPX7y9Nlg//mpUY1mMGVKKH1eUAOCS5habgWc1xpoVQg4Ky4/tf7ZErThSn6z6xryil5IPueM2iDNBm9IRe1CV44pufRHmLhOKOZu5Wf8PeaYcIknmHj8ejaIk2HSAd8m6ZbEaIvJbL/3g5SKNRVIywQ1JkuT2uaOasuZAN8njYGaskt6AVmgklZgctdN5fFhUEo8VzocaXGn7mY4WhmzrooQ2bZsbnqt+D8va+z8Xe64rBsLkm0KzRuBrcLtF+GSa2BWrAOhTPPQK2YLqimz4SP7RMIVU1VFZemIhtJnae6IhZXtOnSt5OLUX5vOMSoZCL+bTRYbMRvlWSFCXHjGgC3UKvlAFuFyB/HowB/2iRa0DrUoN9DKw/ExWdjE71YNBUNEI8CRqzJYaW19QDzyuB/2lt7c0m1yOhqm4+Hx13F88nG7wT30Er1CRyhFb9EJ+owmaIoY+ol+od/oT/QlUtEyWm1Co9425wW6huj7P+S54Fc=</latexit>

conv({xi : i 2 N})

<latexit sha1_base64="/UN5wrAN6tM+iDSmcHAy2eZrM4c=">AAACx3icbVFNaxsxEJW3X6n7Eac99iK6CaQUzK5xaCkUQntpLyWFOgmsFqPVjrMiWmmRtI6N0KHQP9Bf02v7M/pvqrV9cJI+EHq8N6OZ0RSN4MYmyd9edOfuvfsPdh72Hz1+8nR3sPfs1KhWM5gwJZQ+L6gBwSVMLLcCzhsNtC4EnBWXHzv/bA7acCW/2WUDeU0vJJ9xRm2QpoPXpKa20rVjSs79ISZuJRQzt/BT/g5zTLjEXzDx+NV0ECfDZAV8m6QbEqMNTqZ7vR+kVKytQVomqDFZmjQ2d1RbzgT4PmkNNJRd0gvIApW0BpO71VQeHwSlxDOlw5EWr9TtDEdrY5Z1ESK7ls1NrxP/52Wtnb3NHZdNa0GydaFZK7BVuPsiXHINzIplIJRpHnrFrKKaMhs+sk8kXDFV11SWjmgofZbmjlhY2FWHrpNcnPpr0zlGJQPht7NJtRazUZ4VIsSFZwzYQi2S96QKl9uPR/v+oE+0oE2oRbmBTh6Oj0hlE79dNRQMEa0AR67KYKWN9QHxyON+2Ft6c0u3yelomI6HR1/H8fGHzQZ30Av0Eh2iFL1Bx+gTOkETxNBP9Av9Rn+iz5GK5tFiHRr1NjnP0TVE3/8B4EHgVQ==</latexit>

↵̄ = [µ; ⌫]

<latexit sha1_base64="ltYFHPY1fPGLbyrDxAawpT+m2+c=">AAACAHicbVBNS8NAEN34WetX1IMHL4tF8FQSqSiIUPTisYL9gCSUyXbTLt1swu5GKKEX/4oXD4p49Wd489+4bXPQ1gcDj/dmmJkXppwp7Tjf1tLyyuraemmjvLm1vbNr7+23VJJJQpsk4YnshKAoZ4I2NdOcdlJJIQ45bYfD24nffqRSsUQ86FFKgxj6gkWMgDZS1z70Q5A+8HQA+Bp7fpxdYV9kQdeuOFVnCrxI3IJUUIFG1/7yewnJYio04aCU5zqpDnKQmhFOx2U/UzQFMoQ+9QwVEFMV5NMHxvjEKD0cJdKU0Hiq/p7IIVZqFIemMwY9UPPeRPzP8zIdXQY5E2mmqSCzRVHGsU7wJA3cY5ISzUeGAJHM3IrJACQQbTIrmxDc+ZcXSeus6taq5/e1Sv2miKOEjtAxOkUuukB1dIcaqIkIGqNn9IrerCfrxXq3PmatS1Yxc4D+wPr8AVP9lZk=</latexit>

NOW this

2020-06-02 Yao-Liang Yu23

w = rX

i

↵̄iyixi

/X

i2P

µixi �X

i2N

⌫ixi

<latexit sha1_base64="tbUZ/m05DO1k/knSQ19TNgc2lec=">AAACb3icbZHPSisxFMYz4/9er1ZduFAuwaJcF5YZUXQjiG5cSQWrQlPKmTRjg5lMSDJqGWbrA7rzHdz4BmbaWWj1QODLd36HJF8iJbixQfDm+VPTM7Nz8wu1P4t/l5brK6s3Js00ZW2ailTfRWCY4JK1LbeC3SnNIIkEu40ezsv+7SPThqfy2g4V6yZwL3nMKVhn9eovJAE7iOL8qcA7J1hjYrKkxzGJQBMQagBuMyyNinsuyg2p7WCidKpsiscjOSdc4lbhwGwS3/uKXDpETiK9eiNoBqPCP0VYiQaqqtWrv5J+SrOESUsFGNMJA2W7OWjLqWBFjWSGKaAPcM86TkpImOnmo7wKvO2cPo5T7Za0eOR+ncghMWaYRI4s72gme6X5W6+T2fi4m3OpMsskHR8UZwK7lMrwcZ9rRq0YOgFUc3dXTAeggVr3RTUXQjj55J/iZr8ZHjQPrw4ap2dVHPNoA22h/yhER+gUXaAWaiOK3r1Vb8Pb9D78df+fj8eo71Uza+hb+buf5hq71w==</latexit>

r =2

kP

i ↵̄iyixik22

<latexit sha1_base64="DjWuACCqz2igWpudl5dZEnLh7ks=">AAACTHicbVBNa9wwFJS3bZps+rFNj7mILIWcFntJaSkUQnLpMYVuEli55ll+zopIspHkJIvi/5Jf02Pba/9Hb6EQebOHfHRAMJp5D40mr6WwLo7/RL0nT5+tPF9d66+/ePnq9eDNxqGtGsNxwitZmeMcLEqhceKEk3hcGwSVSzzKT/c7/+gMjRWV/ubmNaYKTrQoBQcXpGzwydDPlJUGuB+3nl0y26hMUJaDYSDrGYTLvBMUuFle+os2E+wyG38ft9lgGI/iBehjkizJkCxxkA2uWVHxRqF2XIK10ySuXerBOMEltn3WWKyBn8IJTgPVoNCmfvHHlr4LSkHLyoSjHV2odzc8KGvnKg+TXVT70OvE/3nTxpUfUy903TjU/PahspHUVbQrjBbCIHdyHghwI0JWymcQCnOh1j7TeM4rpUAXnhks2mmSeubwwi0S+k7yw6Tt2koedvOYHI5Hyc7o/ded4e7esrdVskm2yDZJyAeyS76QAzIhnFyRH+QX+R39jP5G19G/29FetNx5S+6ht3IDfDy1Vw==</latexit>

Questions?

2020-06-02 Yao-Liang Yu24