regression part ii - university of...
TRANSCRIPT
![Page 1: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/1.jpg)
CSCI 5521: Paul Schrater
Regression Part II
Note: Several slides taken from tutorialby Bernard Schölkopf
![Page 2: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/2.jpg)
Multi-class Classification• SVM is basically a two-class classifier• One can change the QP formulation to allow multi-
class classification• More commonly, the data set is divided into two parts
“intelligently” in different ways and a separate SVM istrained for each way of division
• Multi-class classification is done by combining theoutput of all the SVM classifiers– Majority rule– Error correcting code– Directed acyclic graph
![Page 3: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/3.jpg)
Epsilon Support Vector Regression (ε-SVR)
• Linear regression in feature space• Unlike in least square regression, the error function is ε
-insensitive loss function– Intuitively, mistake less than ε is ignored
– This leads to sparsity similar to SVM
ε−εValue offtarget
Penalty
Value offtarget
Penalty
Square loss functionε-insensitive loss function
![Page 4: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/4.jpg)
Epsilon Support Vector Regression (ε-SVR)
• Given: a data set {x1, ..., xn} with target values {u1, ...,un}, we want to do ε-SVR
• The optimization problem is
• Similar to SVM, this can be solved as a quadraticprogramming problem
![Page 5: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/5.jpg)
Epsilon Support Vector Regression (ε-SVR)
• C is a parameter to control the amount of influence ofthe error
• The ½||w||2 term serves as controlling the complexity ofthe regression function– This is similar to ridge regression
• After training (solving the QP), we get values of αi andαi
*, which are both zero if xi does not contribute to theerror function
• For a new data z,
![Page 6: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/6.jpg)
CSCI 5521: Paul Schrater
![Page 7: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/7.jpg)
CSCI 5521: Paul Schrater
![Page 8: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/8.jpg)
CSCI 5521: Paul Schrater
![Page 9: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/9.jpg)
CSCI 5521: Paul Schrater
![Page 10: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/10.jpg)
CSCI 5521: Paul Schrater
We want to estimate the noise as well -
Introduce a parameter that bounds the noise and minimize
![Page 11: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/11.jpg)
CSCI 5521: Paul Schrater
![Page 12: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/12.jpg)
CSCI 5521: Paul Schrater
![Page 13: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/13.jpg)
CSCI 5521: Paul Schrater
Illustration
![Page 14: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/14.jpg)
CSCI 5521: Paul Schrater
![Page 15: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/15.jpg)
CSCI 5521: Paul Schrater
![Page 16: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/16.jpg)
CSCI 5521: Paul Schrater
![Page 17: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/17.jpg)
CSCI 5521: Paul Schrater
Handling Heteroscedasticity
![Page 18: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/18.jpg)
CSCI 5521: Paul Schrater
![Page 19: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/19.jpg)
CSCI 5521: Paul Schrater
![Page 20: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/20.jpg)
CSCI 5521: Paul Schrater
More on Kernels
![Page 21: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/21.jpg)
CSCI 5521: Paul Schrater
![Page 22: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/22.jpg)
CSCI 5521: Paul Schrater
![Page 23: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/23.jpg)
CSCI 5521: Paul Schrater
![Page 24: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/24.jpg)
CSCI 5521: Paul Schrater
![Page 25: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/25.jpg)
CSCI 5521: Paul Schrater
![Page 26: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/26.jpg)
CSCI 5521: Paul Schrater
![Page 27: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/27.jpg)
CSCI 5521: Paul Schrater
![Page 28: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/28.jpg)
CSCI 5521: Paul Schrater
![Page 29: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/29.jpg)
CSCI 5521: Paul Schrater
![Page 30: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/30.jpg)
CSCI 5521: Paul Schrater
![Page 31: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/31.jpg)
CSCI 5521: Paul Schrater
![Page 32: Regression Part II - University of Minnesotavision.psych.umn.edu/.../schrater_lab/courses/PattRecog09/Regressi… · Regression Part II Note: Several slides taken from tutorial by](https://reader033.vdocument.in/reader033/viewer/2022050116/5f4d6399d11bf135172c8a93/html5/thumbnails/32.jpg)
CSCI 5521: Paul Schrater