v. linear & nonlinear least-squareskaroger/numci/2016/files/chap05/...3 v.0. examples,...
TRANSCRIPT
![Page 1: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/1.jpg)
1
V. Linear & Nonlinear Least-Squares
V.0.Examples, linear/nonlinear least-squares
V.1.Linear least-squares
V.1.1.Normal equations
V.1.2.The orthogonal transformation method
V.2.Nonlinear least-squares
V.2.1.Newton method
V.2.2.Gauss-Newton method
V. Linear & Nonlinear Least-Squares
![Page 2: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/2.jpg)
2
V.0. Examples, linear/nonlinear least-squares
In practice, one has often to determine unknown parameters of agiven function (from natural laws or model assumptions) through aseries of measurements.
Usually the number of measurements m is much bigger than thenumber of parameters n, i.e.
Measurement
Time
Quantity
Model: Goal: Find1
2
V. Linear & Nonlinear Least-Squares
![Page 3: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/3.jpg)
3
V.0. Examples, linear/nonlinear least-squares
In practice, one has often to determine unknown parameters of agiven function (from natural laws or model assumptions) through aseries of measurements.
Usually the number of measurements m is much bigger than thenumber of parameters n, i.e.
Measurement
Time
Quantity
Model: Goal: Find1
2
Problem: more equations than unknowns! … overdetermined
V. Linear & Nonlinear Least-Squares
![Page 4: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/4.jpg)
4
V.0. Examples, linear/nonlinear least-squares
Goal: Find1
Data:
Linear in model parameters!
Model:
V. Linear & Nonlinear Least-Squares
![Page 5: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/5.jpg)
5
V.0. Examples, linear/nonlinear least-squares
1
Data:
JUSTNOTATION!
Goal: Find
Linear in model parameters!
Model:
V. Linear & Nonlinear Least-Squares
![Page 6: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/6.jpg)
6
V.0. Examples, linear/nonlinear least-squares
● Idea: choose the parameters such that thedistance between the data and the curve isminimal, i.e. the curve that fits best.
V. Linear & Nonlinear Least-Squares
![Page 7: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/7.jpg)
7
V.0. Examples, linear/nonlinear least-squares
● Idea: choose the parameters such that thedistance between the data and the curve isminimal, i.e. the curve that fits best.
Model: 1
V. Linear & Nonlinear Least-Squares
![Page 8: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/8.jpg)
8
V.0. Examples, linear/nonlinear least-squares
● Idea: choose the parameters such that thedistance between the data and the curve isminimal, i.e. the curve that fits best.
MINIMIZE!
V. Linear & Nonlinear Least-Squares
![Page 9: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/9.jpg)
9
V.0. Examples, linear/nonlinear least-squares
● Idea: choose the parameters such that thedistance between the data and the curve isminimal, i.e. the curve that fits best.
● Least squares solution:
Euclidean distance,a.k.a. 2-normDomain of valid parameters
(from application!)
V. Linear & Nonlinear Least-Squares
![Page 10: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/10.jpg)
10
V.0. Examples, linear/nonlinear least-squares
Model: Goal: Find2
Data:
Nonlinear in model parameters!
JUSTNOTATION!
V. Linear & Nonlinear Least-Squares
![Page 11: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/11.jpg)
11
V.0. Examples, linear/nonlinear least-squares
● Idea: choose the parameters such that thedistance between the data and the curve isminimal, i.e. the curve that fits best.
V. Linear & Nonlinear Least-Squares
![Page 12: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/12.jpg)
12
V.0. Examples, linear/nonlinear least-squares
● Idea: choose the parameters such that thedistance between the data and the curve isminimal, i.e. the curve that fits best.
MINIMIZE!
V. Linear & Nonlinear Least-Squares
![Page 13: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/13.jpg)
13
V.0. Examples, linear/nonlinear least-squares
● General problem:
Lin
ear
No
nlin
ear
In parameters!
measurements parameters
Overdetermined!
usually...
V. Linear & Nonlinear Least-Squares
![Page 14: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/14.jpg)
14
V.0. Examples, linear/nonlinear least-squares
● Least squares solution:
● Define scalar-valued function
● Least squares solution:
Linear
Nonlinear
Linear
Nonlinear
V. Linear & Nonlinear Least-Squares
![Page 15: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/15.jpg)
15
V.0. Examples, linear/nonlinear least-squares
● Quiz: linear or nonlinear model?
1)
2) Model parameters:
Model parameters:
V. Linear & Nonlinear Least-Squares
![Page 16: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/16.jpg)
16
V.0. Examples, linear/nonlinear least-squares
● Quiz: linear or nonlinear model?
1)
2) Model parameters:
Model parameters:
V. Linear & Nonlinear Least-Squares
![Page 17: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/17.jpg)
17
V.0. Examples, linear/nonlinear least-squares
● General problem: measurements parameters
Overdetermined!
Least squares solution:
Assumption: has full column rank (linearly indep.)
V. Linear & Nonlinear Least-Squares
![Page 18: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/18.jpg)
18
● Least squares solution:
● Rewrite: Transpose!
Scalar!
~
~
V.1.1. Normal equations
V. Linear & Nonlinear Least-Squares
![Page 19: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/19.jpg)
19
● Gradient of
must vanish at extremum (min/max):
Gradient
(Gauss') Normal equations
Nothing difficult…
~
V.1.1. Normal equations
V. Linear & Nonlinear Least-Squares
![Page 20: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/20.jpg)
20
● Necessary condition:
● Is it a minimum?We have to make sure that the matrix ispositive definite
(Gauss') Normal equations
1
2
Norm (length!)
Unless Because of our assumptionof rank nColumns linearly independent!
Not sufficient!
~
V.1.1. Normal equations
V. Linear & Nonlinear Least-Squares
![Page 21: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/21.jpg)
21
● Geometric interpretation of the normalequations
● It turns out that they lead to a worserconditioned problem, i.e. difficult to solve thenormal equations numerically…Therefore the next method is preferred
.
V.1.1. Normal equations
V. Linear & Nonlinear Least-Squares
![Page 22: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/22.jpg)
22
V.1.2. The orthogonal decomposition method
● Definition:
● Key property:
An orthogonal matrix is a real squarematrix whose columns and rows are orthogonalunit vectors
This means:
Orthogonal matrices leave the Euclidean lengthinvariant
V. Linear & Nonlinear Least-Squares
![Page 23: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/23.jpg)
23
Every matrix with full column rank(the columns are linearly independent) has a so-calledQR-decomposition
where is an orthogonal matrix and an uppertriangular matrix
!Non zero diagonal!
Matlab: [Q,R]=qr(A)
Fact:
V.1.2. The orthogonal decomposition method
V. Linear & Nonlinear Least-Squares
![Page 24: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/24.jpg)
24
Every matrix with full column rank(the columns are linearly independent) has a so-calledQR-decomposition
where is an orthogonal matrix and an uppertriangular matrix
Matlab: [Q,R]=qr(A)
Fact:
Upper triangular
V.1.2. The orthogonal decomposition method
V. Linear & Nonlinear Least-Squares
![Page 25: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/25.jpg)
25
Minimize residuum:
Orthogonal!We still solve the sameminimization problem!!!
V.1.2. The orthogonal decomposition method
V. Linear & Nonlinear Least-Squares
![Page 26: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/26.jpg)
26
Minimize residuum:
Orthogonal!We still solve the sameminimization problem!!!
V.1.2. The orthogonal decomposition method
V. Linear & Nonlinear Least-Squares
![Page 27: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/27.jpg)
27
V.1. Example: linear least-squares
Goal: Find1
Data:
Linear in model parameters!
Model:
V. Linear & Nonlinear Least-Squares
![Page 28: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/28.jpg)
28
V.1. Example: linear least-squares
Goal: Find1
Data:
Model:
Normal equations:
Orthogonal transformation: analogue…
V. Linear & Nonlinear Least-Squares
![Page 29: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/29.jpg)
29
V.2. Nonlinear least-squares
usually...
● General problem: measurements parameters
Overdetermined!
Least squares solution:
V. Linear & Nonlinear Least-Squares
![Page 30: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/30.jpg)
30
● Gradient of
must vanish at extremum (min/max):
Gradient
V.2.1 Newton method
equations
unknowns!
V. Linear & Nonlinear Least-Squares
![Page 31: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/31.jpg)
31
V.2.1 Newton method
Hessian
Gradient
Apply Newton's method to solve
NOT
V. Linear & Nonlinear Least-Squares
![Page 32: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/32.jpg)
32
Linearize the residuum/error equations
V.2.2 Gauss-Newton method
Linearize at some
Linear least squares problem!
V. Linear & Nonlinear Least-Squares
![Page 33: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/33.jpg)
33
V.2.2 Gauss-Newton method
Given an initial guess:
Solve a sequence of linear least squares problems
Until convergence
Small enough
Problem:
...
V. Linear & Nonlinear Least-Squares
![Page 34: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/34.jpg)
34
V.2. Example: nonlinear least-squares
Model: Goal: Find2
Data:
Nonlinear in model parameters!
V. Linear & Nonlinear Least-Squares
![Page 35: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/35.jpg)
35
V.2. Example: nonlinear least-squares
Model: Goal: Find2
Data:
Newton method:
V. Linear & Nonlinear Least-Squares
![Page 36: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/36.jpg)
36
V.2. Example: nonlinear least-squares
Model: Goal: Find2
Data:
Newton method:
Gradient
Two equations in two unknowns!
V. Linear & Nonlinear Least-Squares
![Page 37: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/37.jpg)
37
V.2. Example: nonlinear least-squares
Model: Goal: Find2
Data:
Newton method:
Hessian
V. Linear & Nonlinear Least-Squares
![Page 38: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/38.jpg)
38
V.2. Example: nonlinear least-squares
Model: Goal: Find2
Data:
Newton method:
Iterate!
Initial guess:
V. Linear & Nonlinear Least-Squares
![Page 39: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/39.jpg)
39
V.2. Example: nonlinear least-squares
Model: Goal: Find2
Data:
Gauss-Newton method:
Linearize residuum/error equations:
Linear in parameters now!
V. Linear & Nonlinear Least-Squares
![Page 40: V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of](https://reader033.vdocument.in/reader033/viewer/2022042016/5e7490caa8cd3226a9286ace/html5/thumbnails/40.jpg)
40
V. Summary
● Linear/Nonlinear in parameters
● Overdetermined system of equations!Determine parameters in the least-squares sense...
● Linear:
– Normal equations
– Orthogonal transformation method
– Example
● Nonlinear:
– Newton method
– Gauss-Newton method
– Example
V. Linear & Nonlinear Least-Squares