Download - MATH 685/ CSI 700/ OR 682 Lecture Notes
![Page 1: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/1.jpg)
MATH 685/ CSI 700/ OR 682 Lecture Notes
Lecture 4.
Least squares
![Page 2: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/2.jpg)
Method of least squares
• Measurement errors are inevitable in observational and experimental sciences
• Errors can be smoothed out by averaging over many cases, i.e., taking more measurements than are strictly necessary to determine parameters of system
• Resulting system is overdetermined, so usually there is no exact solution
• In effect, higher dimensional data are projected into lower dimensional space to suppress irrelevant detail
• Such projection is most conveniently accomplished by method of least squares
![Page 3: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/3.jpg)
Linear least squares
![Page 4: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/4.jpg)
Data fitting
![Page 5: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/5.jpg)
Data fitting
![Page 6: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/6.jpg)
Example
![Page 7: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/7.jpg)
Example
![Page 8: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/8.jpg)
Example
![Page 9: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/9.jpg)
Existence/Uniqueness
![Page 10: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/10.jpg)
Normal Equations
![Page 11: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/11.jpg)
Orthogonality
![Page 12: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/12.jpg)
Orthogonality
![Page 13: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/13.jpg)
Orthogonal Projector
![Page 14: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/14.jpg)
Pseudoinverse
![Page 15: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/15.jpg)
Sensitivity and Conditioning
![Page 16: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/16.jpg)
Sensitivity and Conditioning
![Page 17: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/17.jpg)
Solving normal equations
![Page 18: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/18.jpg)
Example
![Page 19: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/19.jpg)
Example
![Page 20: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/20.jpg)
Shortcomings
![Page 21: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/21.jpg)
Augmented system method
![Page 22: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/22.jpg)
Augmented system method
![Page 23: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/23.jpg)
Orthogonal Transformations
![Page 24: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/24.jpg)
Triangular Least Squares
![Page 25: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/25.jpg)
Triangular Least Squares
![Page 26: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/26.jpg)
QR Factorization
![Page 27: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/27.jpg)
Orthogonal Bases
![Page 28: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/28.jpg)
Computing QR factorization To compute QR factorization of m × n matrix A, with m > n, we annihilate subdiagonal entries of successive columns of A, eventually reaching upper triangular form
Similar to LU factorization by Gaussian elimination, but use orthogonal transformations instead of elementary elimination
matrices
Possible methods include Householder transformations Givens rotations Gram-Schmidt orthogonalization
![Page 29: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/29.jpg)
Householder Transformation
![Page 30: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/30.jpg)
Example
![Page 31: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/31.jpg)
Householder QR factorization
![Page 32: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/32.jpg)
Householder QR factorization
![Page 33: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/33.jpg)
Householder QR factorization For solving linear least squares problem, product Q ofHouseholder transformations need not be formed explicitly
R can be stored in upper triangle of array initiallycontaining A
Householder vectors v can be stored in (now zero) lowertriangular portion of A (almost)
Householder transformations most easily applied in thisform anyway
![Page 34: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/34.jpg)
Example
![Page 35: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/35.jpg)
Example
![Page 36: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/36.jpg)
Example
![Page 37: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/37.jpg)
Example
![Page 38: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/38.jpg)
Givens Rotations
![Page 39: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/39.jpg)
Givens Rotations
![Page 40: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/40.jpg)
Example
![Page 41: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/41.jpg)
Givens QR factorization
![Page 42: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/42.jpg)
Givens QR factorization Straightforward implementation of Givens method requiresabout 50% more work than Householder method, and alsorequires more storage, since each rotation requires twonumbers, c and s, to define it
These disadvantages can be overcome, but requires morecomplicated implementation
Givens can be advantageous for computing QRfactorization when many entries of matrix are already zero,since those annihilations can then be skipped
![Page 43: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/43.jpg)
Gram-Schmidt orthogonalization
![Page 44: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/44.jpg)
Gram-Schmidt algorithm
![Page 45: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/45.jpg)
Modified Gram-Schmidt
![Page 46: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/46.jpg)
Modified Gram-Schmidt QR factorization
![Page 47: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/47.jpg)
Rank Deficiency If rank(A) < n, then QR factorization still exists, but yieldssingular upper triangular factor R, and multiple vectors xgive minimum residual norm
Common practice selects minimum residual solution xhaving smallest norm
Can be computed by QR factorization with column pivotingor by singular value decomposition (SVD)
Rank of matrix is often not clear cut in practice, so relativetolerance is used to determine rank
![Page 48: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/48.jpg)
Near Rank Deficiency
![Page 49: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/49.jpg)
QR with Column Pivoting
![Page 50: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/50.jpg)
QR with Column Pivoting
![Page 51: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/51.jpg)
Singular Value Decomposition
![Page 52: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/52.jpg)
Example: SVD
![Page 53: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/53.jpg)
Applications of SVD
![Page 54: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/54.jpg)
Pseudoinverse
![Page 55: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/55.jpg)
Orthogonal Bases
![Page 56: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/56.jpg)
Lower-rank Matrix Approximation
![Page 57: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/57.jpg)
Total Least Squares Ordinary least squares is applicable when right-hand
side b is subject to random error but matrix A is known accurately
When all data, including A, are subject to error, then total least squares is more appropriate
Total least squares minimizes orthogonal distances, rather than vertical distances, between model and data
Total least squares solution can be computed from SVD of [A, b]
![Page 58: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/58.jpg)
Comparison of Methods
![Page 59: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/59.jpg)
Comparison of Methods
![Page 60: MATH 685/ CSI 700/ OR 682 Lecture Notes](https://reader036.vdocument.in/reader036/viewer/2022062500/568159a5550346895dc700bc/html5/thumbnails/60.jpg)
Comparison of Methods