least squares and data fitting - university of illinoisย ยท data fitting -not always a line fit!...
TRANSCRIPT
![Page 1: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/1.jpg)
Least Squares and Data Fitting
![Page 2: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/2.jpg)
How do we best fit a set of data points?
Data fitting
![Page 3: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/3.jpg)
Given๐ data points { ๐ก!, ๐ฆ! , โฆ , ๐ก", ๐ฆ" }, we want to find the function ๐ฆ = ๐ฅ! + ๐ฅ" ๐ก
that best fit the data (or better, we want to find the coefficients ๐ฅ#, ๐ฅ!).
Thinking geometrically, we can think โwhat is the line that most nearly passes through all the points?โ
Linear Least Squares 1) Fitting with a line
![Page 4: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/4.jpg)
Given๐ data points { ๐ก!, ๐ฆ! , โฆ , ๐ก", ๐ฆ" }, we want to find ๐ฅ# and ๐ฅ!such that
๐ฆ$ = ๐ฅ# + ๐ฅ! ๐ก$ โ๐ โ 1,๐
or in matrix form: Note that this system of linear equations has more equations than unknowns โOVERDETERMINED SYSTEMS๐ร๐ ๐ร๐ ๐ร๐
1 ๐ก!โฎ โฎ1 ๐ก"
๐ฅ#๐ฅ! =
๐ฆ!โฎ๐ฆ"
๐จ ๐ = ๐
We want to find the appropriate linear combination of the columns of ๐จthat makes up the vector ๐.
If a solution exists that satisfies ๐จ ๐ = ๐ then ๐ โ ๐๐๐๐๐(๐จ)
![Page 5: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/5.jpg)
Linear Least Squaresโข In most cases, ๐ โ ๐๐๐๐๐(๐จ) and ๐จ ๐ = ๐ does not have an
exact solution!
โข Therefore, an overdetermined system is better expressed as
๐จ ๐ โ ๐
![Page 6: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/6.jpg)
Linear Least Squaresโข Least Squares: find the solution ๐ that minimizes the residual
๐ = ๐ โ ๐จ ๐
โข Letโs define the function ๐ as the square of the 2-norm of the residual
๐ ๐ = ๐ โ ๐จ ๐ %%
![Page 7: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/7.jpg)
Linear Least Squaresโข Least Squares: find the solution ๐ that minimizes the residual
๐ = ๐ โ ๐จ ๐
โข Letโs define the function ๐ as the square of the 2-norm of the residual
๐ ๐ = ๐ โ ๐จ ๐ %%
โข Then the least squares problem becomesmin๐๐ (๐)
โข Suppose ๐:โ" โ โ is a smooth function, then ๐ ๐ reaches a (local) maximum or minimum at a point ๐โ โ โ" only if
โ๐ ๐โ = 0
![Page 8: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/8.jpg)
How to find the minimizer?โข To minimize the 2-norm of the residual vector
min๐๐ ๐ = ๐ โ ๐จ ๐ %
%
๐ ๐ = (๐ โ ๐จ ๐)((๐ โ ๐จ ๐)
โ๐ ๐ = 2(๐จ( ๐ โ ๐จ(๐จ ๐)
First order necessary condition:โ๐ ๐ = 0 โ ๐จ( ๐ โ ๐จ(๐จ ๐ = ๐ โ ๐จ(๐จ ๐ = ๐จ( ๐
Second order sufficient condition:๐ท%๐ ๐ = 2๐จ(๐จ2๐จ(๐จ is a positive semi-definite matrix โ the solution is a minimum
Normal Equations โ solve a linear system of equations
![Page 9: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/9.jpg)
Linear Least Squares (another approach)โข Find ๐ = ๐จ ๐ which is closest to the vector ๐โข What is the vector ๐ = ๐จ ๐ โ ๐๐๐๐๐(๐จ) that is closest to vector ๐ in
the Euclidean norm?
When ๐ = ๐ โ ๐ = ๐ โ ๐จ ๐ is orthogonal to all columns of ๐จ, then ๐ is closest to ๐
๐จ๐ป๐ = ๐จ๐ป(๐ โ ๐จ ๐)=0 ๐จ"๐จ ๐ = ๐จ" ๐
![Page 10: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/10.jpg)
Summary:โข ๐จ is a ๐ร๐ matrix, where ๐ > ๐. โข ๐ is the number of data pair points. ๐ is the number of parameters of the
โbest fitโ function.
โข Linear Least Squares problem ๐จ ๐ โ ๐ always has solution.
โข The Linear Least Squares solution ๐ minimizes the square of the 2-norm of the residual:
min๐
๐ โ ๐จ ๐ %%
โข One method to solve the minimization problem is to solve the system of Normal Equations
๐จ(๐จ ๐ = ๐จ( ๐
โข Letโs see some examples and discuss the limitations of this method.
![Page 11: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/11.jpg)
Example:
Solve: ๐จ#๐จ ๐ = ๐จ# ๐
![Page 12: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/12.jpg)
โข Does not need to be a line! For example, here we are fitting the data using a quadratic curve.
Data fitting - not always a line fit!
Linear Least Squares: The problem is linear in its coefficients!
![Page 13: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/13.jpg)
Another examplesWe want to find the coefficients of the quadratic function that best fits the data points:
We would not want our โfitโ curve to pass through the data points exactly as we are looking to model the general trend and not capture the noise.
๐ฆ = ๐ฅ! + ๐ฅ" ๐ก + ๐ฅ# ๐ก#
![Page 14: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/14.jpg)
Data fitting
1 ๐ก! ๐ก!"โฎ โฎ โฎ1 ๐ก# ๐ก#"
๐ฅ$๐ฅ!๐ฅ"
=๐ฆ!โฎ๐ฆ#
(๐ก$,๐ฆ$)
Solve: ๐จ#๐จ ๐ = ๐จ# ๐
![Page 15: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/15.jpg)
Which function is not suitable for linear least squares?
A) ๐ฆ = ๐ + ๐ ๐ฅ + ๐ ๐ฅ# + ๐ ๐ฅ%B) ๐ฆ = ๐ฅ ๐ + ๐ ๐ฅ + ๐ ๐ฅ# + ๐ ๐ฅ%C) ๐ฆ = ๐ sin ๐ฅ + ๐/ cos ๐ฅD) ๐ฆ = ๐ sin ๐ฅ + ๐ฅ/ cos ๐๐ฅE) ๐ฆ = ๐ ๐&#' + ๐ ๐#'
![Page 16: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/16.jpg)
Computational Cost๐จ#๐จ ๐ = ๐จ# ๐
โข Compute ๐จ(๐จ: ๐ ๐๐%
โข Factorize ๐จ(๐จ: LU โ ๐ %2๐
2 , Cholesky โ๐ !2๐
2
โข Solve ๐ ๐%โข Since ๐ > ๐ the overall cost is ๐ ๐๐%
![Page 17: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/17.jpg)
Short questionsGiven the data in the table below, which of the plots shows the line of best fit in terms of least squares?
A) B)
C) D)
![Page 18: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/18.jpg)
Short questionsGiven the data in the table below, and the least squares model
๐ฆ = ๐! + ๐% sin ๐ก๐ + ๐2 sin ๐ก๐/2 + ๐3 sin ๐ก๐/4
written in matrix form as
determine the entry ๐ด%2 of the matrix ๐จ. Note that indices start with 1.
A) โ1.0B) 1.0C) โ 0.7D) 0.7E) 0.0
![Page 19: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/19.jpg)
Solving Linear Least Squares with SVD
![Page 20: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/20.jpg)
๐จ is a ๐ร๐ matrix where ๐ > ๐(more points to fit than coefficient to be determined)
Normal Equations: ๐จ!๐จ ๐ = ๐จ! ๐
โข The solution ๐จ ๐ โ ๐ is unique if and only if ๐๐๐๐ ๐ = ๐(๐จ is full column rank)
โข ๐๐๐๐ ๐ = ๐ โ columns of ๐จ are linearly independent โ ๐ non-zero singular values โ ๐จ! ๐จ has only positive eigenvalues โ ๐จ!๐จ is a symmetricand positive definite matrix โ ๐จ!๐จ is invertible
๐ = ๐จ!๐จ "๐๐จ! ๐
โข If ๐๐๐๐ ๐ < ๐, then ๐จ is rank-deficient, and solution of linear least squares problem is not unique.
What we have learned so farโฆ
![Page 21: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/21.jpg)
Condition number for Normal EquationsFinding the least square solution of ๐จ ๐ โ ๐ (where ๐จ is full rank matrix) using the Normal Equations
๐จ(๐จ ๐ = ๐จ( ๐
has some advantages, since we are solving a square system of linear equations with a symmetric matrix (and hence it is possible to use decompositions such as Cholesky Factorization)
However, the normal equations tend to worsen the conditioning of the matrix.
๐๐๐๐ ๐จ(๐จ = (๐๐๐๐ ๐จ )%
How can we solve the least square problem without squaring the condition of the matrix?
![Page 22: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/22.jpg)
SVD to solve linear least squares problems
We want to find the least square solution of ๐จ ๐ โ ๐, where ๐จ = ๐ผ ๐บ ๐ฝ๐ป
or better expressed in reduced form: ๐จ = ๐ผ5 ๐บ๐น ๐ฝ๐ป
๐จ =โฎ โฆ โฎ๐# โฆ ๐$โฎ โฆ โฎ
๐#โฑ
๐%0โฎ0
โฆ ๐ฏ#" โฆโฎ โฎ โฎโฆ ๐ฏ%" โฆ
๐จ is a๐ร๐ rectangular matrix where ๐ > ๐, and hence the SVD decomposition is given by:
![Page 23: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/23.jpg)
Recall Reduced SVD
๐จ = ๐ผ4 ๐บ๐น ๐ฝ๐ป
๐ร๐ ๐ร๐๐ร๐
๐ร๐
๐ > ๐
![Page 24: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/24.jpg)
![Page 25: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/25.jpg)
SVD to solve linear least squares problems
We want to find the least square solution of ๐จ ๐ โ ๐, where ๐จ = ๐ผ$ ๐บ๐น ๐ฝ๐ป
Normal equations: ๐จ"๐จ ๐ = ๐จ" ๐ โถ ๐ผ& ๐บ๐น ๐ฝ" " ๐ผ& ๐บ๐น ๐ฝ" ๐ = ๐ผ& ๐บ๐น ๐ฝ" "๐
๐จ =โฎ โฆ โฎ๐! โฆ ๐Aโฎ โฆ โฎ
๐!โฑ
๐A
โฆ ๐ฏ!( โฆโฎ โฎ โฎโฆ ๐ฏA( โฆ
๐จ = ๐ผ4 ๐บ๐น ๐ฝ๐ป
๐ฝ ๐บ๐น๐ผ&" ๐ผ& ๐บ๐น ๐ฝ" ๐ = ๐ฝ ๐บ๐น๐ผ&"๐
๐ฝ ๐บ๐น ๐บ๐น ๐ฝ"๐ = ๐ฝ ๐บ๐น๐ผ&"๐
๐บ๐น ( ๐ฝ"๐ = ๐บ๐น๐ผ&"๐ When can we take the inverse of the singular matrix?
![Page 26: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/26.jpg)
๐บ๐น " ๐ฝ#๐ = ๐บ๐น๐ผ$#๐
1) Full rank matrix (๐$ โ 0 โ๐):
๐ฝ#๐ = ๐บ๐น %&๐ผ$#๐๐ = ๐ฝ ๐บ๐น "'๐ผ$! ๐
Unique solution:
๐ร๐ ๐ร๐ ๐ร๐๐ร1๐ร1
rank ๐จ = ๐
2) Rank deficient matrix ( rank ๐จ = ๐ < ๐ )
Solution is not unique!!
Find solution ๐ such thatmin๐๐ ๐ = ๐ โ ๐จ ๐ "
"
and also
๐บ๐น " ๐ฝ#๐ = ๐บ๐น๐ผ$#๐
min๐
๐ ๐
![Page 27: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/27.jpg)
2) Rank deficient matrix (continue)
Change of variables: Set ๐ฝ#๐ = ๐ and then solve ๐บ๐น ๐ = ๐ผ$#๐ for the variable ๐
๐#โฑ
๐)0
โฑ0
๐ฆ#โฎ๐ฆ)๐ฆ)*#โฎ๐ฆ%
=
๐#"๐โฎ
๐)"๐๐)*#" ๐โฎ
๐%"๐
๐ฆ+ =๐+"๐๐+
๐ = 1,2, โฆ , ๐
What do we do when ๐ > ๐? Which choice of ๐ฆ+ will minimize
๐ ๐ = ๐ฝ ๐ ๐?
๐ฆ+ = 0, ๐ = ๐ + 1,โฆ , ๐Set
We want to find the solution ๐ that satisfies ๐บ๐น " ๐ฝ#๐ = ๐บ๐น๐ผ$#๐ and also satisfies min๐
๐ ๐
Evaluate
๐ = ๐ฝ๐ =โฎ โฆ โฎ๐& โฆ ๐)โฎ โฆ โฎ
๐ฆ&๐ฆ"โฎ๐ฆ)
๐ =9*+&
)
๐ฆ* ๐๐ = 9*+&-!./
)(๐*#๐)๐*
๐๐
![Page 28: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/28.jpg)
Solving Least Squares Problem with SVD (summary)
โข Find ๐ that satisfies min๐
๐ โ ๐จ ๐ ))
โข Find ๐ that satisfiesmin๐
๐บ๐น ๐ โ ๐ผ$#๐ ""
โข Propose ๐ that is solution of ๐บ๐น ๐ = ๐ผ5(๐
โข Evaluate: ๐ = ๐ผ5(๐
โข Set: ๐ฆ$ = hO"P", if ๐$ โ 0
0, otherwise๐ = 1,โฆ , ๐
โข Then compute ๐ = ๐ฝ ๐
Cost:
๐ ๐
๐
๐#
Cost of SVD:๐(๐ ๐#)
![Page 29: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/29.jpg)
โข If ๐$โ 0 for โ๐ = 1,โฆ , ๐, then the solution ๐ = ๐ฝ ๐บ๐น Q!๐ผ5( ๐ is unique (and not a โchoiceโ).
โข If at least one of the singular values is zero, then the proposed solution ๐ is the one with the smallest 2-norm ( ๐ % is minimal ) that minimizes the 2-norm of the residual ๐บ๐น ๐ โ ๐ผ5(๐ %
โข Since ๐ % = ๐ฝ ๐ %= ๐ %, then the solution ๐ is also the one with the smallest 2-norm ( ๐ % is minimal ) for all possible ๐ for which ๐จ๐ โ ๐ % is minimal.
Solving Least Squares Problem with SVD (summary)
![Page 30: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/30.jpg)
Solve ๐จ ๐ โ ๐ or ๐ผ> ๐บ๐น๐ฝ๐ป๐ โ ๐
๐ โ ๐ฝ ๐บ๐น A ๐ผ>B ๐
Solving Least Squares Problem with SVD (summary)
![Page 31: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/31.jpg)
Consider solving the least squares problem ๐จ ๐ โ ๐, where the singular value decomposition of the matrix ๐จ = ๐ผ ๐บ ๐ฝ๐ป๐ is:
Determine ๐ โ ๐จ ๐ (
Example:
![Page 32: Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit! LinearLeast Squares: The problem is linear in its coefficients! Another examples We](https://reader035.vdocument.in/reader035/viewer/2022071012/5fca66658340766a8023baa7/html5/thumbnails/32.jpg)
ExampleSuppose you have ๐จ = ๐ผ ๐บ ๐ฝ๐ป๐ calculated. What is the cost of solving
min๐
๐ โ ๐จ ๐ %% ?
A) ๐(๐)B) ๐( ๐%)C) ๐(๐๐)D) ๐ ๐E) ๐( ๐%)