![Page 1: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/1.jpg)
Linear Regression
CSC321: Intro to Machine Learning and Neural Networks, Winter 2016
Michael Guerzhoy
Slides from:
Andrew Ng
https://xkcd.com/1007/
![Page 2: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/2.jpg)
Notation:m = Number of training examplesx’s = “input” variable / featuresy’s = “output” variable / “target” variable
Size in feet2 (x) Price ($) in 1000's (y)
2104 4601416 2321534 315852 178… …
Training set ofhousing prices(Portland, OR)
![Page 3: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/3.jpg)
0
100
200
300
400
500
0 500 1000 1500 2000 2500 3000
0
100
200
300
400
500
0 500 1000 1500 2000 2500 3000
Housing Prices(Portland, OR)
Price(in 1000s of dollars)
Size (feet2)
Supervised Learning
Given the “right answer” for each example in the data.
Regression Problem
Predict real-valued output
![Page 4: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/4.jpg)
Training Set
Learning Algorithm
hSize of house
Estimated price
How do we represent h ?
Linear regression with one variable.Univariate linear regression.
![Page 5: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/5.jpg)
How to choose ‘s ?
Training Set
Hypothesis:
‘s: Parameters
Size in feet2 (x) Price ($) in 1000's (y)
2104 4601416 2321534 315852 178… …
![Page 6: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/6.jpg)
0
1
2
3
0 1 2 3
0
1
2
3
0 1 2 3
0
1
2
3
0 1 2 3
![Page 7: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/7.jpg)
y
x
Idea: Choose so that is close to for our
training examples
![Page 8: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/8.jpg)
Quadratic cost function – on the board
![Page 9: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/9.jpg)
0
1
2
3
0 1 2 3
y
x
(for fixed , this is a function of x) (function of the parameter , for a fixed 𝜃0)
0
1
2
3
-0.5 0 0.5 1 1.5 2 2.5
𝐽𝜃0(𝜃1)
![Page 10: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/10.jpg)
Hypothesis:
Parameters:
Cost Function:
Goal:
![Page 11: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/11.jpg)
Cost Function Surface Plot
![Page 12: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/12.jpg)
Contour Plots• For a function F(x, y) of two variables, assigned
different colours to different values of F• Pick some values to plot• The result will be contours – curves in the graph
along which the values of F(x, y) are constant
![Page 13: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/13.jpg)
(for fixed , this is a function of x) (function of the parameters )
![Page 14: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/14.jpg)
(for fixed , this is a function of x) (function of the parameters )
Cost Function Contour Plot
![Page 15: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/15.jpg)
(for fixed , this is a function of x) (function of the parameters )
![Page 16: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/16.jpg)
Have some function
Want
Outline:
• Start with some
• Keep changing to reduce
until we hopefully end up at a minimum
![Page 17: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/17.jpg)
Gradient Descent on the board
![Page 18: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/18.jpg)
1
0
J(0,1)
![Page 19: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/19.jpg)
0
1
J(0,1)
![Page 20: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/20.jpg)
For Linear Regression, J is bowl-shaped (“convex”)
![Page 21: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/21.jpg)
(for fixed , this is a function of x) (function of the parameters )
![Page 22: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/22.jpg)
(for fixed , this is a function of x) (function of the parameters )
![Page 23: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/23.jpg)
(for fixed , this is a function of x) (function of the parameters )
![Page 24: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/24.jpg)
(for fixed , this is a function of x) (function of the parameters )
![Page 25: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/25.jpg)
(for fixed , this is a function of x) (function of the parameters )
![Page 26: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/26.jpg)
(for fixed , this is a function of x) (function of the parameters )
![Page 27: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/27.jpg)
(for fixed , this is a function of x) (function of the parameters )
![Page 28: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/28.jpg)
(for fixed , this is a function of x) (function of the parameters )
![Page 29: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/29.jpg)
(for fixed , this is a function of x) (function of the parameters )
![Page 30: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/30.jpg)
Orange: y = 1Blue: y = 0
Linear Regression vs. k-Nearest Neighbours
![Page 31: Linear Regression - cs.toronto.edu · Linear Regression vs. k-Nearest Neighbours •Linear Regression: the boundary can only be linear •Nearest Neighbours: the boundary can more](https://reader035.vdocument.in/reader035/viewer/2022062402/5fc4af926159526a8b20fcff/html5/thumbnails/31.jpg)
Linear Regression vs. k-Nearest Neighbours
• Linear Regression: the boundary can only be linear
• Nearest Neighbours: the boundary can more complex
• Which is better?• Depends on what the actual boundary looks like
• Depends on whether we have enough data to figure out the correct complex boundary