perceptron - virginia techcourses.cs.vt.edu/cs5824/fall15/pdfs/5-perceptron.pdf · perceptron...
TRANSCRIPT
![Page 1: Perceptron - Virginia Techcourses.cs.vt.edu/cs5824/Fall15/pdfs/5-Perceptron.pdf · Perceptron Summary • Linear classifier multiplies weights by input features • Learn by updating](https://reader030.vdocument.in/reader030/viewer/2022040622/5d388ca188c9931d5c8c3254/html5/thumbnails/1.jpg)
Perceptron
Machine Learning CSx824/ECEx242
Bert Huang Virginia Tech
![Page 2: Perceptron - Virginia Techcourses.cs.vt.edu/cs5824/Fall15/pdfs/5-Perceptron.pdf · Perceptron Summary • Linear classifier multiplies weights by input features • Learn by updating](https://reader030.vdocument.in/reader030/viewer/2022040622/5d388ca188c9931d5c8c3254/html5/thumbnails/2.jpg)
Plan
• Perceptron
• Multi-class perceptron
![Page 3: Perceptron - Virginia Techcourses.cs.vt.edu/cs5824/Fall15/pdfs/5-Perceptron.pdf · Perceptron Summary • Linear classifier multiplies weights by input features • Learn by updating](https://reader030.vdocument.in/reader030/viewer/2022040622/5d388ca188c9931d5c8c3254/html5/thumbnails/3.jpg)
Perceptron• AKA single-layer neural network
• linear classifier
• online learner
f (x) = sign
X
i
wixi
!
x1 x2 x3 x4 x5
w1w2 w3w4 w5
y
=
(+1 if
Pi wixi � 0
�1 ifP
i wixi < 0
![Page 4: Perceptron - Virginia Techcourses.cs.vt.edu/cs5824/Fall15/pdfs/5-Perceptron.pdf · Perceptron Summary • Linear classifier multiplies weights by input features • Learn by updating](https://reader030.vdocument.in/reader030/viewer/2022040622/5d388ca188c9931d5c8c3254/html5/thumbnails/4.jpg)
Online Training
• Get example x and oracle label y
• Try current classifier f(x)
• If f(x) = y, celebrate
• If f(x) ≠ y, fix it
f(x) = sign(w1x1 + w2x2 + w3x4 + … )
if y = +1 and f(x) = -1, w•x is too smallif y = -1 and f(x) = +1, w•x is too big
![Page 5: Perceptron - Virginia Techcourses.cs.vt.edu/cs5824/Fall15/pdfs/5-Perceptron.pdf · Perceptron Summary • Linear classifier multiplies weights by input features • Learn by updating](https://reader030.vdocument.in/reader030/viewer/2022040622/5d388ca188c9931d5c8c3254/html5/thumbnails/5.jpg)
How to Fix Your Broken Perceptron
y = -1f(x) = sign(w1 x1 + w2 x2) = +1
2(3) + 1(1)
w ← w - x if f(x) = +1 and y = -1w ← w + x if f(x) = -1 and y = +1w ← w if f(x) = y
-2(3) + 0(1)
w ← w + yx if f(x) ≠ yW1 = 2 - 3W2 = 1 - 1
![Page 6: Perceptron - Virginia Techcourses.cs.vt.edu/cs5824/Fall15/pdfs/5-Perceptron.pdf · Perceptron Summary • Linear classifier multiplies weights by input features • Learn by updating](https://reader030.vdocument.in/reader030/viewer/2022040622/5d388ca188c9931d5c8c3254/html5/thumbnails/6.jpg)
How to Fix Your Broken Perceptronw ← w - x if f(x) = +1 and y = -1w ← w + x if f(x) = -1 and y = +1w ← w if f(x) = y
w ← w + yx if f(x) ≠ y
y = +1, f(x) = -1w • x (w + x) • x
w • x + x • x
(w - x) • x
w • x - x • x
≥y = -1, f(x) = +1
w • x
≤
![Page 7: Perceptron - Virginia Techcourses.cs.vt.edu/cs5824/Fall15/pdfs/5-Perceptron.pdf · Perceptron Summary • Linear classifier multiplies weights by input features • Learn by updating](https://reader030.vdocument.in/reader030/viewer/2022040622/5d388ca188c9931d5c8c3254/html5/thumbnails/7.jpg)
![Page 8: Perceptron - Virginia Techcourses.cs.vt.edu/cs5824/Fall15/pdfs/5-Perceptron.pdf · Perceptron Summary • Linear classifier multiplies weights by input features • Learn by updating](https://reader030.vdocument.in/reader030/viewer/2022040622/5d388ca188c9931d5c8c3254/html5/thumbnails/8.jpg)
Perceptron Learning
w ← w + αyx if f(x) ≠ y α = learning rate
b ← b + αy if f(x) ≠ y
f(x) = sign(w • x)sign(w • x + b)
![Page 9: Perceptron - Virginia Techcourses.cs.vt.edu/cs5824/Fall15/pdfs/5-Perceptron.pdf · Perceptron Summary • Linear classifier multiplies weights by input features • Learn by updating](https://reader030.vdocument.in/reader030/viewer/2022040622/5d388ca188c9931d5c8c3254/html5/thumbnails/9.jpg)
Multiclass Output
+1-1
![Page 10: Perceptron - Virginia Techcourses.cs.vt.edu/cs5824/Fall15/pdfs/5-Perceptron.pdf · Perceptron Summary • Linear classifier multiplies weights by input features • Learn by updating](https://reader030.vdocument.in/reader030/viewer/2022040622/5d388ca188c9931d5c8c3254/html5/thumbnails/10.jpg)
Multiclass Output
![Page 11: Perceptron - Virginia Techcourses.cs.vt.edu/cs5824/Fall15/pdfs/5-Perceptron.pdf · Perceptron Summary • Linear classifier multiplies weights by input features • Learn by updating](https://reader030.vdocument.in/reader030/viewer/2022040622/5d388ca188c9931d5c8c3254/html5/thumbnails/11.jpg)
Multiclass Perceptron• Separate weights for each class: w0, w1, w2, w3, w4, w5, w6, …
• Predict according to maximum scoring class:
f(x) = argmax wy'•x• Learning update:
wy ← wy + x wf(x) ← wf(x) - x
y'
![Page 12: Perceptron - Virginia Techcourses.cs.vt.edu/cs5824/Fall15/pdfs/5-Perceptron.pdf · Perceptron Summary • Linear classifier multiplies weights by input features • Learn by updating](https://reader030.vdocument.in/reader030/viewer/2022040622/5d388ca188c9931d5c8c3254/html5/thumbnails/12.jpg)
Multiclass Perceptron• Learning update:
wy ← wy + x wf(x) ← wf(x) - xx
f(x) = 6
w2 ← w2 + x (make w2 • x greater)y = 2
w6 ← w6 - x (make w6 • x lesser)
![Page 13: Perceptron - Virginia Techcourses.cs.vt.edu/cs5824/Fall15/pdfs/5-Perceptron.pdf · Perceptron Summary • Linear classifier multiplies weights by input features • Learn by updating](https://reader030.vdocument.in/reader030/viewer/2022040622/5d388ca188c9931d5c8c3254/html5/thumbnails/13.jpg)
Multiclass Perceptron• Learning update:
wy ← wy + x wf(x) ← wf(x) - xx
f(x) = 2
w2 ← w2 + xy = 2
w2 ← w2 - x
![Page 14: Perceptron - Virginia Techcourses.cs.vt.edu/cs5824/Fall15/pdfs/5-Perceptron.pdf · Perceptron Summary • Linear classifier multiplies weights by input features • Learn by updating](https://reader030.vdocument.in/reader030/viewer/2022040622/5d388ca188c9931d5c8c3254/html5/thumbnails/14.jpg)
Multiclass Perceptron• Learning update:
wy ← wy + x wf(x) ← wf(x) - xx
f(x) = 2 w2 ← w2 + x - xy = 2 = w2
![Page 15: Perceptron - Virginia Techcourses.cs.vt.edu/cs5824/Fall15/pdfs/5-Perceptron.pdf · Perceptron Summary • Linear classifier multiplies weights by input features • Learn by updating](https://reader030.vdocument.in/reader030/viewer/2022040622/5d388ca188c9931d5c8c3254/html5/thumbnails/15.jpg)
Perceptron Summary• Linear classifier multiplies weights by input features
• Learn by updating when wrong
• If score too low, make score higher
• If score too high, make score lower
• Multiclass: use multiple weight vectors, choose max scoring
• Learn by adjusting score of predicted class and true class