lecture 02 linear algebra basics - github pages
TRANSCRIPT
![Page 1: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/1.jpg)
Linear Algebra Basics
Machine Learning CS 4641
Nakul Gopalan
Georgia Tech
These slides are based on slides from Le Song and Andres Mendez-Vazquez, Chao Zhang, Mahdi Roozbahani, Rodrigo Valente
![Page 2: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/2.jpg)
Some logistics
• Creating team.
• Office hours are started from next week.
• First quiz out this Thursday.
• First assignment out this Thursday (early release).
![Page 3: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/3.jpg)
Outline
• Linear Algebra Basics
• Norms
• Multiplications
• Matrix Inversion
• Trace and Determinant
• Eigen Values and Eigen Vectors
• Singular Value Decomposition
• Matrix Calculus
![Page 4: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/4.jpg)
Why Linear Algebra?
𝑛 × 𝑑𝑛 𝑑
𝑑𝑑
𝑑
𝑑
1
![Page 5: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/5.jpg)
Example
![Page 6: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/6.jpg)
Linear Algebra Basics
𝑛 × 𝑑 𝑑 × 𝑛
𝑑 × 𝑑
![Page 7: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/7.jpg)
Outline
• Linear Algebra Basics
• Norms
• Multiplications
• Matrix Inversion
• Trace and Determinant
• Eigen Values and Eigen Vectors
• Singular Value Decomposition
![Page 8: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/8.jpg)
Norms
𝑑
𝑑
𝑑
𝑑
𝑑
![Page 9: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/9.jpg)
Norms
10
𝑑
𝑑𝑛
![Page 10: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/10.jpg)
Vector Norm Examples
![Page 11: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/11.jpg)
Special Matrices
𝑑 × 𝑑
𝑑
𝑑 × 𝑑
𝑑
![Page 12: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/12.jpg)
Outline
• Linear Algebra Basics
• Norms
• Multiplications
• Matrix Inversion
• Trace and Determinant
• Eigen Values and Eigen Vectors
• Singular Value Decomposition
![Page 13: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/13.jpg)
Multiplications
𝑛 × 𝑑 𝑑 × 𝑝
𝑛 × 𝑝 𝑑
𝑑
𝑑
𝑑 𝑛
𝑥𝑦𝑇
𝑥𝑇𝑦: 𝑥 ⊗ 𝑦
𝑥𝑦𝑇
![Page 14: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/14.jpg)
Multiplications
T𝑥 ⊗ 𝑦 =
![Page 15: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/15.jpg)
Inner Product Properties
![Page 16: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/16.jpg)
Inner Product Properties
![Page 17: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/17.jpg)
Inner Product Properties
![Page 18: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/18.jpg)
Example
![Page 19: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/19.jpg)
Matrix multiplication geometric meaning
![Page 20: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/20.jpg)
Outline
• Linear Algebra Basics
• Norms
• Multiplications
• Matrix Inversion
• Trace and Determinant
• Eigen Values and Eigen Vectors
• Matrix Decomposition
![Page 21: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/21.jpg)
Linear Independence and Matrix Rank
𝑑𝑑
𝑑
𝑑
𝑛 × 𝑑
![Page 22: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/22.jpg)
Matrix Rank: Examples
What are the ranks for the following matrices?
![Page 23: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/23.jpg)
Geometric meaning
![Page 24: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/24.jpg)
Matrix Inverse
𝑑 × 𝑑
![Page 25: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/25.jpg)
Outline
• Linear Algebra Basics
• Norms
• Multiplications
• Matrix Inversion
• Trace and Determinant
• Eigen Values and Eigen Vectors
• Singular Value Decomposition
![Page 26: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/26.jpg)
Matrix Trace
𝑑 × 𝑑
𝑑 × 𝑑
𝑑 × 𝑑
𝑑 × 𝑑
𝑑
![Page 27: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/27.jpg)
Matrix Determinant
𝑑
![Page 28: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/28.jpg)
Properties of Matrix Determinant
![Page 29: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/29.jpg)
Outline
• Linear Algebra Basics
• Norms
• Multiplications
• Matrix Inversion
• Trace and Determinant
• Eigen Values and Eigen Vectors
• Singular Value Decomposition
![Page 30: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/30.jpg)
Eigenvalues and Eigenvectors
𝑑 × 𝑑𝑑
![Page 31: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/31.jpg)
Computing Eigenvalues and Eigenvectors
𝑑.
𝑑 𝑑
(𝐴 − 𝜆𝐼)
(𝐴 − 𝜆𝐼) (𝐴 − 𝜆𝐼)
(𝐴 − 𝜆𝐼)
![Page 32: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/32.jpg)
Eigenvalue Example
Slide credit: Shubham Kumbhar
![Page 33: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/33.jpg)
Matrix Eigen Decomposition
columns
𝑑
𝑑
![Page 34: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/34.jpg)
Outline
• Linear Algebra Basics
• Norms
• Multiplications
• Matrix Inversion
• Trace and Determinant
• Eigen Values and Eigen Vectors
• Singular Value Decomposition
![Page 35: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/35.jpg)
Covariance matrix
![Page 36: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/36.jpg)
Covariance matrix
![Page 37: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/37.jpg)
Correlation matrix
![Page 38: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/38.jpg)
Correlation matrix
![Page 39: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/39.jpg)
Singular Value Decomposition
43
𝑋𝑛×𝑑n: instancesd: dimensionsX is a centered matrix
𝑋 = 𝑈Σ𝑉𝑇
𝑈𝑛×𝑛 → 𝑢𝑛𝑖𝑡𝑎𝑟𝑦 𝑚𝑎𝑡𝑟𝑖𝑥 → 𝑈 × 𝑈𝑇 = 𝐼
Σ𝑛×𝑑 → 𝑑𝑖𝑎𝑔𝑜𝑛𝑎𝑙 𝑚𝑎𝑡𝑟𝑖𝑥
V𝑑×𝑑 → 𝑢𝑛𝑖𝑡𝑎𝑟𝑦 𝑚𝑎𝑡𝑟𝑖𝑥 → 𝑉 × 𝑉𝑇 = 𝐼
=
ddd
d
dd
nn
n
vv
vv
uu
uu
X
.........
...
...
......
.........
000
000
00
00
00
.........
...
...
......
.........
1
11111
11
111
𝑈 Σ 𝑉𝑇
𝑑 < 𝑛
![Page 40: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/40.jpg)
𝐶𝑑×𝑑 =𝑋𝑇𝑋
𝑛
𝑋 = 𝑈Σ𝑉𝑇
𝐶 =𝑋𝑇𝑋
𝑛
𝐶 =𝑉Σ𝑇𝑈𝑇𝑈Σ𝑉𝑇
𝑛=𝑉Σ2𝑉𝑇
𝑛
Covariance matrix:
![Page 41: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/41.jpg)
𝐶 =𝑉Σ2𝑉𝑇
𝑛= 𝑉
Σ2
𝑛𝑉𝑇
So, we can directly calculate eigenvalue of a covariance matrix by having the singular value of matrix X directly
𝜆𝑖 =Σ𝑖2
𝑛➔ The eigenvalues of covariance matrix
According to Eigen-decomposition definition ➔𝐶𝑉 = VΛ
𝐶𝑉 = 𝑉Σ2
𝑛𝑉𝑇𝑉 = 𝑉
Σ2
𝑛
𝜆𝑖: Eigenvalue of 𝐶 or covariance matrix
Σ𝑖: Singular value of 𝑋 matrix
![Page 42: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/42.jpg)
Geometric Meaning of SVD
Image Credit: Kevin Binz
![Page 43: Lecture 02 Linear Algebra Basics - GitHub Pages](https://reader033.vdocument.in/reader033/viewer/2022042104/62568244ba0b375986057dc6/html5/thumbnails/43.jpg)
Summary
• Linear Algebra Basics
• Norms
• Multiplications
• Matrix Inversion
• Trace and Determinant
• Eigen Values and Eigen Vectors
• Singular Value Decomposition