![Page 1: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/1.jpg)
IntroductionLinear Regression
Calculus in AI and Machine Learning 1
Qiyam Tung
G-TEAMSDepartment of Computer Science
Univeristy of Arizona
November 10, 2011
Qiyam Tung Calculus and AI 1
![Page 2: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/2.jpg)
IntroductionLinear Regression
Introduction
Linear Regression is one of the tools we use in machine learning.
Predicting house prices asa function of its size.
Predicting consumptionspending (a large numberof input variables).
Predicting the effect offactories on environments.
Qiyam Tung Calculus and AI 1
![Page 3: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/3.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
For each problem, find the parameters for the line given a set ofpoints. If the line doesn’t exist, explain why not.
1 (1,1) (2,2)
2 (2,2) (4, 10)
3 (1,7)
4 (1,1) (2,3) (3,2) (4,4)
Qiyam Tung Calculus and AI 1
![Page 4: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/4.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
Figure: (1,1) (2,2)
Qiyam Tung Calculus and AI 1
![Page 5: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/5.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
Figure: (2,2) (4,10)
Qiyam Tung Calculus and AI 1
![Page 6: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/6.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
Figure: (1.7)
Qiyam Tung Calculus and AI 1
![Page 7: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/7.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
Figure: (1,1) (2,3) (3,2) (4,4)
Qiyam Tung Calculus and AI 1
![Page 8: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/8.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
How do you estimate a line?
Figure: When we fit a line, we want to minimize the average distance ofthe points to the line we’re fitting. The distance is indicated by thevertical lines from the point to the line.
Qiyam Tung Calculus and AI 1
![Page 9: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/9.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
Error =1
m
m∑i=1
(hθ(x (i)) − y (i))2 (1)
Scary!
Hold on...
Qiyam Tung Calculus and AI 1
![Page 10: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/10.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
Error =1
m
m∑i=1
(hθ(x (i)) − y (i))2 (1)
Scary!
Hold on...
Qiyam Tung Calculus and AI 1
![Page 11: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/11.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
Error =1
m
m∑i=1
(hθ(x (i)) − y (i))2 (1)
Scary!
Hold on...
Qiyam Tung Calculus and AI 1
![Page 12: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/12.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
hθ(x) − y = (θ0 + θ1x) − y (2)
Distance from line?
Qiyam Tung Calculus and AI 1
![Page 13: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/13.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
(hθ(x) − y)2 = ((θ0 + θ1x) − y)2 (3)
Squaring it always gives us a positive error. We don’t want theerrors to cancel out.
Qiyam Tung Calculus and AI 1
![Page 14: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/14.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
We want to minimize the average distance of the points to theline we’re fitting. The distance/error is indicated by the verticallines from the point to the line.
Error =1
m
m∑i=1
(hθ(x (i)) − y (i))2 (4)
Average: 1m
“Distance” or mean-squared-error: (hθ(x (i)) − y (i))2
Input: x (1), x (2), ..., x (m)
Line: hθ(x (i))
Output: y (1), y (2), ..., y (m)
Qiyam Tung Calculus and AI 1
![Page 15: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/15.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
We want to minimize the average distance of the points to theline we’re fitting. The distance/error is indicated by the verticallines from the point to the line.
Error =1
m
m∑i=1
(hθ(x (i)) − y (i))2 (4)
Average: 1m
“Distance” or mean-squared-error: (hθ(x (i)) − y (i))2
Input: x (1), x (2), ..., x (m)
Line: hθ(x (i))
Output: y (1), y (2), ..., y (m)
Qiyam Tung Calculus and AI 1
![Page 16: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/16.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
We want to minimize the average distance of the points to theline we’re fitting. The distance/error is indicated by the verticallines from the point to the line.
Error =1
m
m∑i=1
(hθ(x (i)) − y (i))2 (4)
Average: 1m
“Distance” or mean-squared-error: (hθ(x (i)) − y (i))2
Input: x (1), x (2), ..., x (m)
Line: hθ(x (i))
Output: y (1), y (2), ..., y (m)
Qiyam Tung Calculus and AI 1
![Page 17: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/17.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
We want to minimize the average distance of the points to theline we’re fitting. The distance/error is indicated by the verticallines from the point to the line.
Error =1
m
m∑i=1
(hθ(x (i)) − y (i))2 (4)
Average: 1m
“Distance” or mean-squared-error: (hθ(x (i)) − y (i))2
Input: x (1), x (2), ..., x (m)
Line: hθ(x (i))
Output: y (1), y (2), ..., y (m)
Qiyam Tung Calculus and AI 1
![Page 18: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/18.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
We want to minimize the average distance of the points to theline we’re fitting. The distance/error is indicated by the verticallines from the point to the line.
Error =1
m
m∑i=1
(hθ(x (i)) − y (i))2 (4)
Average: 1m
“Distance” or mean-squared-error: (hθ(x (i)) − y (i))2
Input: x (1), x (2), ..., x (m)
Line: hθ(x (i))
Output: y (1), y (2), ..., y (m)
Qiyam Tung Calculus and AI 1
![Page 19: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/19.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
We want to minimize the average distance of the points to theline we’re fitting. The distance/error is indicated by the verticallines from the point to the line.
Error =1
m
m∑i=1
(hθ(x (i)) − y (i))2 (4)
Average: 1m
“Distance” or mean-squared-error: (hθ(x (i)) − y (i))2
Input: x (1), x (2), ..., x (m)
Line: hθ(x (i))
Output: y (1), y (2), ..., y (m)
Qiyam Tung Calculus and AI 1
![Page 20: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/20.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
Error =1
4
4∑i=1
(hθ(x (i)) − y (i))2 (5)
Error =1
4
((θ0 + θ1x
(1) − y (1))2 +
(θ0 + θ1x(2) − y (2))2 +
(θ0 + θ1x(3) − y (3))2 +
(θ0 + θ1x(4) − y (4))2
)
Error =1
4((θ0+θ1−1)2+(θ0+2θ1−3)2+(θ0+3θ1−2)2+(θ0+4θ1−4)2)
(6)
Qiyam Tung Calculus and AI 1
![Page 21: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/21.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
Error =1
4
4∑i=1
(hθ(x (i)) − y (i))2 (5)
Error =1
4
((θ0 + θ1x
(1) − y (1))2 +
(θ0 + θ1x(2) − y (2))2 +
(θ0 + θ1x(3) − y (3))2 +
(θ0 + θ1x(4) − y (4))2
)
Error =1
4((θ0+θ1−1)2+(θ0+2θ1−3)2+(θ0+3θ1−2)2+(θ0+4θ1−4)2)
(6)
Qiyam Tung Calculus and AI 1
![Page 22: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/22.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
Error =1
4
4∑i=1
(hθ(x (i)) − y (i))2 (5)
Error =1
4
((θ0 + θ1x
(1) − y (1))2 +
(θ0 + θ1x(2) − y (2))2 +
(θ0 + θ1x(3) − y (3))2 +
(θ0 + θ1x(4) − y (4))2
)
Error =1
4((θ0+θ1−1)2+(θ0+2θ1−3)2+(θ0+3θ1−2)2+(θ0+4θ1−4)2)
(6)
Qiyam Tung Calculus and AI 1
![Page 23: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/23.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
Given the data (x (1), y (1)), (x (2), y (2)), (x (3), y (3)), (x (4), y (4)) or(1,1), (2,3), (3,2), (4,4), find the error of each of the lines with thegiven parameters.
1 θ0 = 2, θ1 = 0
2 θ0 = 0, θ1 = 1
Qiyam Tung Calculus and AI 1
![Page 24: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/24.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
Figure: As we decrease the slope, the error increases dramatically
Qiyam Tung Calculus and AI 1
![Page 25: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/25.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
Figure: Similarly, if we increase the slope the error also increasesdramatically
Qiyam Tung Calculus and AI 1
![Page 26: Calculus in AI and Machine Learning 1 - University of Arizonaime.math.arizona.edu/g-teams/Profiles/QT/Calc/ai... · 2011-11-13 · Calculus in AI and Machine Learning 1 Qiyam Tung](https://reader033.vdocument.in/reader033/viewer/2022042314/5f020df27e708231d402597d/html5/thumbnails/26.jpg)
IntroductionLinear Regression
Exact LinesEstimating LinesMinimization
Figure: The error plotted as a function of theta.
How do you minimize the function? In other words, how do youfind the minimum y for this error function?
Qiyam Tung Calculus and AI 1