03 machine learning linear algebra
TRANSCRIPT
![Page 1: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/1.jpg)
Machine Learning for Data MiningLinear Algebra Review
Andres Mendez-Vazquez
May 14, 2015
1 / 50
![Page 2: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/2.jpg)
Outline
1 IntroductionWhat is a Vector?
2 Vector SpacesDefinitionLinear Independence and Basis of Vector SpacesNorm of a VectorInner ProductMatricesTrace and DeterminantMatrix DecompositionSingular Value Decomposition
2 / 50
![Page 3: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/3.jpg)
Outline
1 IntroductionWhat is a Vector?
2 Vector SpacesDefinitionLinear Independence and Basis of Vector SpacesNorm of a VectorInner ProductMatricesTrace and DeterminantMatrix DecompositionSingular Value Decomposition
3 / 50
![Page 4: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/4.jpg)
What is a Vector?
A ordered tuple of numbers
x =
x1x2...
xn
Expressing a magnitude and a direction
4 / 50
![Page 5: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/5.jpg)
What is a Vector?
A ordered tuple of numbers
x =
x1x2...
xn
Expressing a magnitude and a direction
Magnitude
Direction
4 / 50
![Page 6: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/6.jpg)
Outline
1 IntroductionWhat is a Vector?
2 Vector SpacesDefinitionLinear Independence and Basis of Vector SpacesNorm of a VectorInner ProductMatricesTrace and DeterminantMatrix DecompositionSingular Value Decomposition
5 / 50
![Page 7: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/7.jpg)
Vector Spaces
DefinitionA vector is an element of a vector space
Vector Space VIt is a set that contains all linear combinations of its elements:
1 If x,y ∈ V then x + y ∈ V .2 If x ∈ V then αx ∈ V for any scalar α.3 There exists 0 ∈ V then x + 0 = x for any x ∈ V .
A subspaceIt is a subset of a vector space that is also a vector space
6 / 50
![Page 8: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/8.jpg)
Vector Spaces
DefinitionA vector is an element of a vector space
Vector Space VIt is a set that contains all linear combinations of its elements:
1 If x,y ∈ V then x + y ∈ V .2 If x ∈ V then αx ∈ V for any scalar α.3 There exists 0 ∈ V then x + 0 = x for any x ∈ V .
A subspaceIt is a subset of a vector space that is also a vector space
6 / 50
![Page 9: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/9.jpg)
Vector Spaces
DefinitionA vector is an element of a vector space
Vector Space VIt is a set that contains all linear combinations of its elements:
1 If x,y ∈ V then x + y ∈ V .2 If x ∈ V then αx ∈ V for any scalar α.3 There exists 0 ∈ V then x + 0 = x for any x ∈ V .
A subspaceIt is a subset of a vector space that is also a vector space
6 / 50
![Page 10: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/10.jpg)
Vector Spaces
DefinitionA vector is an element of a vector space
Vector Space VIt is a set that contains all linear combinations of its elements:
1 If x,y ∈ V then x + y ∈ V .2 If x ∈ V then αx ∈ V for any scalar α.3 There exists 0 ∈ V then x + 0 = x for any x ∈ V .
A subspaceIt is a subset of a vector space that is also a vector space
6 / 50
![Page 11: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/11.jpg)
Vector Spaces
DefinitionA vector is an element of a vector space
Vector Space VIt is a set that contains all linear combinations of its elements:
1 If x,y ∈ V then x + y ∈ V .2 If x ∈ V then αx ∈ V for any scalar α.3 There exists 0 ∈ V then x + 0 = x for any x ∈ V .
A subspaceIt is a subset of a vector space that is also a vector space
6 / 50
![Page 12: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/12.jpg)
Classic Example
Euclidean Space R3
7 / 50
![Page 13: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/13.jpg)
Span
DefinitionThe span of any set of vectors {x1,x2, ...,xn} is defined as:
span (x1,x2, ...,xn) = α1x1 + α2x2 + ...+ αnxn
What Examples can you Imagine?Give it a shot!!!
8 / 50
![Page 14: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/14.jpg)
Span
DefinitionThe span of any set of vectors {x1,x2, ...,xn} is defined as:
span (x1,x2, ...,xn) = α1x1 + α2x2 + ...+ αnxn
What Examples can you Imagine?Give it a shot!!!
8 / 50
![Page 15: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/15.jpg)
Subspaces of Rn
A line through the origin in Rn
A plane in Rn
9 / 50
![Page 16: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/16.jpg)
Subspaces of Rn
A line through the origin in Rn
A plane in Rn
9 / 50
![Page 17: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/17.jpg)
Outline
1 IntroductionWhat is a Vector?
2 Vector SpacesDefinitionLinear Independence and Basis of Vector SpacesNorm of a VectorInner ProductMatricesTrace and DeterminantMatrix DecompositionSingular Value Decomposition
10 / 50
![Page 18: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/18.jpg)
Linear Independence and Basis of Vector Spaces
Fact 1A vector x is a linearly independent of a set of vectors {x1,x2, ...,xn} if itdoes not lie in their span.
Fact 2A set of vectors is linearly independent if every vector is linearlyindependent of the rest.
The Rest1 A basis of a vector space V is a linearly independent set of vectors
whose span is equal to V2 If the basis has d vectors then the vector space V has dimensionality
d.
11 / 50
![Page 19: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/19.jpg)
Linear Independence and Basis of Vector Spaces
Fact 1A vector x is a linearly independent of a set of vectors {x1,x2, ...,xn} if itdoes not lie in their span.
Fact 2A set of vectors is linearly independent if every vector is linearlyindependent of the rest.
The Rest1 A basis of a vector space V is a linearly independent set of vectors
whose span is equal to V2 If the basis has d vectors then the vector space V has dimensionality
d.
11 / 50
![Page 20: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/20.jpg)
Linear Independence and Basis of Vector Spaces
Fact 1A vector x is a linearly independent of a set of vectors {x1,x2, ...,xn} if itdoes not lie in their span.
Fact 2A set of vectors is linearly independent if every vector is linearlyindependent of the rest.
The Rest1 A basis of a vector space V is a linearly independent set of vectors
whose span is equal to V2 If the basis has d vectors then the vector space V has dimensionality
d.
11 / 50
![Page 21: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/21.jpg)
Outline
1 IntroductionWhat is a Vector?
2 Vector SpacesDefinitionLinear Independence and Basis of Vector SpacesNorm of a VectorInner ProductMatricesTrace and DeterminantMatrix DecompositionSingular Value Decomposition
12 / 50
![Page 22: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/22.jpg)
Norm of a Vector
DefinitionA norm ‖u‖ measures the magnitud of the vector.
Properties1 Homogeneity: ‖αx‖ = α ‖x‖.2 Triangle inequality: ‖x + y‖ ≤ ‖x‖+ ‖y‖.3 Point Separation ‖x‖ = 0 if and only if x = 0.
Examples1 Manhattan or `1-norm : ‖x‖1 =
∑di=1 |xi |.
2 Euclidean or `2-norm : ‖x‖2 =√∑d
i=1 x2i .
13 / 50
![Page 23: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/23.jpg)
Norm of a Vector
DefinitionA norm ‖u‖ measures the magnitud of the vector.
Properties1 Homogeneity: ‖αx‖ = α ‖x‖.2 Triangle inequality: ‖x + y‖ ≤ ‖x‖+ ‖y‖.3 Point Separation ‖x‖ = 0 if and only if x = 0.
Examples1 Manhattan or `1-norm : ‖x‖1 =
∑di=1 |xi |.
2 Euclidean or `2-norm : ‖x‖2 =√∑d
i=1 x2i .
13 / 50
![Page 24: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/24.jpg)
Norm of a Vector
DefinitionA norm ‖u‖ measures the magnitud of the vector.
Properties1 Homogeneity: ‖αx‖ = α ‖x‖.2 Triangle inequality: ‖x + y‖ ≤ ‖x‖+ ‖y‖.3 Point Separation ‖x‖ = 0 if and only if x = 0.
Examples1 Manhattan or `1-norm : ‖x‖1 =
∑di=1 |xi |.
2 Euclidean or `2-norm : ‖x‖2 =√∑d
i=1 x2i .
13 / 50
![Page 25: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/25.jpg)
Norm of a Vector
DefinitionA norm ‖u‖ measures the magnitud of the vector.
Properties1 Homogeneity: ‖αx‖ = α ‖x‖.2 Triangle inequality: ‖x + y‖ ≤ ‖x‖+ ‖y‖.3 Point Separation ‖x‖ = 0 if and only if x = 0.
Examples1 Manhattan or `1-norm : ‖x‖1 =
∑di=1 |xi |.
2 Euclidean or `2-norm : ‖x‖2 =√∑d
i=1 x2i .
13 / 50
![Page 26: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/26.jpg)
Norm of a Vector
DefinitionA norm ‖u‖ measures the magnitud of the vector.
Properties1 Homogeneity: ‖αx‖ = α ‖x‖.2 Triangle inequality: ‖x + y‖ ≤ ‖x‖+ ‖y‖.3 Point Separation ‖x‖ = 0 if and only if x = 0.
Examples1 Manhattan or `1-norm : ‖x‖1 =
∑di=1 |xi |.
2 Euclidean or `2-norm : ‖x‖2 =√∑d
i=1 x2i .
13 / 50
![Page 27: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/27.jpg)
Norm of a Vector
DefinitionA norm ‖u‖ measures the magnitud of the vector.
Properties1 Homogeneity: ‖αx‖ = α ‖x‖.2 Triangle inequality: ‖x + y‖ ≤ ‖x‖+ ‖y‖.3 Point Separation ‖x‖ = 0 if and only if x = 0.
Examples1 Manhattan or `1-norm : ‖x‖1 =
∑di=1 |xi |.
2 Euclidean or `2-norm : ‖x‖2 =√∑d
i=1 x2i .
13 / 50
![Page 28: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/28.jpg)
ExamplesExample `1-norm and `2-norm
14 / 50
![Page 29: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/29.jpg)
Outline
1 IntroductionWhat is a Vector?
2 Vector SpacesDefinitionLinear Independence and Basis of Vector SpacesNorm of a VectorInner ProductMatricesTrace and DeterminantMatrix DecompositionSingular Value Decomposition
15 / 50
![Page 30: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/30.jpg)
Inner Product
DefinitionThe inner product between u and v
〈u, v〉 =n∑
i=1uivi .
It is the projection of one vector onto the other one
Remark: It is related to the Euclidean norm: 〈u,u〉 = ‖u‖22.
16 / 50
![Page 31: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/31.jpg)
Inner ProductDefinitionThe inner product between u and v
〈u, v〉 =n∑
i=1uivi .
It is the projection of one vector onto the other one
Remark: It is related to the Euclidean norm: 〈u,u〉 = ‖u‖22.16 / 50
![Page 32: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/32.jpg)
Properties
MeaningThe inner product is a measure of correlation between two vectors, scaledby the norms of the vectors
if u · v > 0, u and v are aligned
17 / 50
![Page 33: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/33.jpg)
Properties
MeaningThe inner product is a measure of correlation between two vectors, scaledby the norms of the vectors
if u · v > 0, u and v are aligned
17 / 50
![Page 34: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/34.jpg)
Properties
The inner product is a measure of correlation between two vectors,scaled by the norms of the vectors
18 / 50
![Page 35: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/35.jpg)
Properties
The inner product is a measure of correlation between two vectors,scaled by the norms of the vectors
19 / 50
![Page 36: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/36.jpg)
Properties
The inner product is a measure of correlation between two vectors,scaled by the norms of the vectors
20 / 50
![Page 37: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/37.jpg)
Definitions involving the norm
OrthonormalThe vectors in orthonormal basis have unit Euclidean norm and areorthonorgonal.
To express a vector x in an orthonormal basisFor example, given x = α1b1 + α2b2
〈x, b1〉 = 〈α1b1 + α2b2, b1〉= α1 〈b1, b1〉+ α2 〈b2, b1〉= α1 + 0
Likewise, 〈x, b2〉 = α2
21 / 50
![Page 38: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/38.jpg)
Definitions involving the norm
OrthonormalThe vectors in orthonormal basis have unit Euclidean norm and areorthonorgonal.
To express a vector x in an orthonormal basisFor example, given x = α1b1 + α2b2
〈x, b1〉 = 〈α1b1 + α2b2, b1〉= α1 〈b1, b1〉+ α2 〈b2, b1〉= α1 + 0
Likewise, 〈x, b2〉 = α2
21 / 50
![Page 39: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/39.jpg)
Definitions involving the norm
OrthonormalThe vectors in orthonormal basis have unit Euclidean norm and areorthonorgonal.
To express a vector x in an orthonormal basisFor example, given x = α1b1 + α2b2
〈x, b1〉 = 〈α1b1 + α2b2, b1〉= α1 〈b1, b1〉+ α2 〈b2, b1〉= α1 + 0
Likewise, 〈x, b2〉 = α2
21 / 50
![Page 40: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/40.jpg)
Outline
1 IntroductionWhat is a Vector?
2 Vector SpacesDefinitionLinear Independence and Basis of Vector SpacesNorm of a VectorInner ProductMatricesTrace and DeterminantMatrix DecompositionSingular Value Decomposition
22 / 50
![Page 41: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/41.jpg)
Linear Operator
DefinitionA linear operator L : U → V is a map from a vector space U to anothervector space V satisfies:
L (u1 + u2) = L (u1) + L (u2)
Something NotableIf the dimension n of U and m of V are finite, L can be represented bym × n matrix:
A =
a11 a12 · · · a1na21 a22 · · · a2n
· · ·am1 am2 · · · amn
23 / 50
![Page 42: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/42.jpg)
Linear Operator
DefinitionA linear operator L : U → V is a map from a vector space U to anothervector space V satisfies:
L (u1 + u2) = L (u1) + L (u2)
Something NotableIf the dimension n of U and m of V are finite, L can be represented bym × n matrix:
A =
a11 a12 · · · a1na21 a22 · · · a2n
· · ·am1 am2 · · · amn
23 / 50
![Page 43: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/43.jpg)
Thus, product of
The product of two linear operator can be seen as the multiplicationof two matrices
AB =
a11 a12 · · · a1na21 a22 · · · a2n
· · ·am1 am2 · · · amn
b11 b12 · · · b1pb21 b22 · · · b2p
· · ·bn1 bn2 · · · bnp
=
∑n
i=1 a1ibi1∑n
i=1 a1ibi2 · · ·∑n
i=1 a1ibip∑ni=1 a2ibi1
∑ni=1 a2ibi2 · · ·
∑ni=1 a2ibip
· · ·∑ni=1 amibi1
∑ni=1 amibi2 · · ·
∑ni=1 amibip
Note: if A is m × n and B is n × p, then AB is m × p.
24 / 50
![Page 44: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/44.jpg)
Thus, product of
The product of two linear operator can be seen as the multiplicationof two matrices
AB =
a11 a12 · · · a1na21 a22 · · · a2n
· · ·am1 am2 · · · amn
b11 b12 · · · b1pb21 b22 · · · b2p
· · ·bn1 bn2 · · · bnp
=
∑n
i=1 a1ibi1∑n
i=1 a1ibi2 · · ·∑n
i=1 a1ibip∑ni=1 a2ibi1
∑ni=1 a2ibi2 · · ·
∑ni=1 a2ibip
· · ·∑ni=1 amibi1
∑ni=1 amibi2 · · ·
∑ni=1 amibip
Note: if A is m × n and B is n × p, then AB is m × p.
24 / 50
![Page 45: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/45.jpg)
Transpose of a MatrixThe transpose of a matrix is obtained by flipping the rows andcolumns
AT =
a11 a21 · · · an1a12 a22 · · · an2
· · ·a1m a2m · · · anm
Which the following properties(AT)T
= A
(A + B)T = AT + BT
(AB)T = BT AT
Not only that, we have the inner product
〈u, v〉 = uT v25 / 50
![Page 46: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/46.jpg)
Transpose of a MatrixThe transpose of a matrix is obtained by flipping the rows andcolumns
AT =
a11 a21 · · · an1a12 a22 · · · an2
· · ·a1m a2m · · · anm
Which the following properties(AT)T
= A
(A + B)T = AT + BT
(AB)T = BT AT
Not only that, we have the inner product
〈u, v〉 = uT v25 / 50
![Page 47: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/47.jpg)
Transpose of a MatrixThe transpose of a matrix is obtained by flipping the rows andcolumns
AT =
a11 a21 · · · an1a12 a22 · · · an2
· · ·a1m a2m · · · anm
Which the following properties(AT)T
= A
(A + B)T = AT + BT
(AB)T = BT AT
Not only that, we have the inner product
〈u, v〉 = uT v25 / 50
![Page 48: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/48.jpg)
Transpose of a MatrixThe transpose of a matrix is obtained by flipping the rows andcolumns
AT =
a11 a21 · · · an1a12 a22 · · · an2
· · ·a1m a2m · · · anm
Which the following properties(AT)T
= A
(A + B)T = AT + BT
(AB)T = BT AT
Not only that, we have the inner product
〈u, v〉 = uT v25 / 50
![Page 49: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/49.jpg)
Transpose of a MatrixThe transpose of a matrix is obtained by flipping the rows andcolumns
AT =
a11 a21 · · · an1a12 a22 · · · an2
· · ·a1m a2m · · · anm
Which the following properties(AT)T
= A
(A + B)T = AT + BT
(AB)T = BT AT
Not only that, we have the inner product
〈u, v〉 = uT v25 / 50
![Page 50: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/50.jpg)
As always, we have the identity operator
The identity operator in matrix multiplication is defined as
I =
1 0 · · · 00 1 · · · 0
· · ·0 0 · · · 1
With propertiesFor any matrix A, AI = A.I is the identity operator for the matrix product.
26 / 50
![Page 51: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/51.jpg)
As always, we have the identity operator
The identity operator in matrix multiplication is defined as
I =
1 0 · · · 00 1 · · · 0
· · ·0 0 · · · 1
With propertiesFor any matrix A, AI = A.I is the identity operator for the matrix product.
26 / 50
![Page 52: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/52.jpg)
As always, we have the identity operator
The identity operator in matrix multiplication is defined as
I =
1 0 · · · 00 1 · · · 0
· · ·0 0 · · · 1
With propertiesFor any matrix A, AI = A.I is the identity operator for the matrix product.
26 / 50
![Page 53: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/53.jpg)
Column Space, Row Space and Rank
Let A be an m × n matrixWe have the following spaces...
Column spaceSpan of the columns of A.Linear subspace of Rm .
Row spaceSpan of the rows of A.Linear subspace of Rn .
27 / 50
![Page 54: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/54.jpg)
Column Space, Row Space and Rank
Let A be an m × n matrixWe have the following spaces...
Column spaceSpan of the columns of A.Linear subspace of Rm .
Row spaceSpan of the rows of A.Linear subspace of Rn .
27 / 50
![Page 55: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/55.jpg)
Column Space, Row Space and Rank
Let A be an m × n matrixWe have the following spaces...
Column spaceSpan of the columns of A.Linear subspace of Rm .
Row spaceSpan of the rows of A.Linear subspace of Rn .
27 / 50
![Page 56: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/56.jpg)
Column Space, Row Space and Rank
Let A be an m × n matrixWe have the following spaces...
Column spaceSpan of the columns of A.Linear subspace of Rm .
Row spaceSpan of the rows of A.Linear subspace of Rn .
27 / 50
![Page 57: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/57.jpg)
Important facts
Something NotableThe column and row space of any matrix have the same dimension.
The rankThe dimension is the rank of the matrix.
28 / 50
![Page 58: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/58.jpg)
Important facts
Something NotableThe column and row space of any matrix have the same dimension.
The rankThe dimension is the rank of the matrix.
28 / 50
![Page 59: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/59.jpg)
Range and Null Space
RangeSet of vectors equal to Au for some u ∈ Rn .
Range (A) = {x|x = Au for some u ∈ Rn}
It is a linear subspace of Rm and also called the column space of A.
Null SpaceWe have the following definition
Null Space (A) = {u|Au = 0}
It is a linear subspace of Rm .
29 / 50
![Page 60: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/60.jpg)
Range and Null Space
RangeSet of vectors equal to Au for some u ∈ Rn .
Range (A) = {x|x = Au for some u ∈ Rn}
It is a linear subspace of Rm and also called the column space of A.
Null SpaceWe have the following definition
Null Space (A) = {u|Au = 0}
It is a linear subspace of Rm .
29 / 50
![Page 61: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/61.jpg)
Range and Null Space
RangeSet of vectors equal to Au for some u ∈ Rn .
Range (A) = {x|x = Au for some u ∈ Rn}
It is a linear subspace of Rm and also called the column space of A.
Null SpaceWe have the following definition
Null Space (A) = {u|Au = 0}
It is a linear subspace of Rm .
29 / 50
![Page 62: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/62.jpg)
Range and Null Space
RangeSet of vectors equal to Au for some u ∈ Rn .
Range (A) = {x|x = Au for some u ∈ Rn}
It is a linear subspace of Rm and also called the column space of A.
Null SpaceWe have the following definition
Null Space (A) = {u|Au = 0}
It is a linear subspace of Rm .
29 / 50
![Page 63: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/63.jpg)
Important fact
Something NotableEvery vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.
30 / 50
![Page 64: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/64.jpg)
Important fact
Something NotableEvery vector in the null space is orthogonal to the rows of A.The null space and row space of a matrix are orthogonal.
30 / 50
![Page 65: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/65.jpg)
Range and Column Space
We have another interpretation of the matrix-vector product
Au = (A1 A2 · · · An)
u1u2...
un
=u1A1 + u2A2 + · · ·+ unAn
ThusThe result is a linear combination of the columns of A.Actually, the range is the column space.
31 / 50
![Page 66: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/66.jpg)
Range and Column Space
We have another interpretation of the matrix-vector product
Au = (A1 A2 · · · An)
u1u2...
un
=u1A1 + u2A2 + · · ·+ unAn
ThusThe result is a linear combination of the columns of A.Actually, the range is the column space.
31 / 50
![Page 67: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/67.jpg)
Range and Column Space
We have another interpretation of the matrix-vector product
Au = (A1 A2 · · · An)
u1u2...
un
=u1A1 + u2A2 + · · ·+ unAn
ThusThe result is a linear combination of the columns of A.Actually, the range is the column space.
31 / 50
![Page 68: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/68.jpg)
Matrix Inverse
Something NotableFor an n × n matrix A: rank + dim(null space) = n.if dim(null space)= 0 then A is full rank.In this case, the action of the matrix is invertible.The inversion is also linear and consequently can be represented byanother matrix A−1.A−1 is the only matrix such that A−1A = AA−1 = I .
32 / 50
![Page 69: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/69.jpg)
Matrix Inverse
Something NotableFor an n × n matrix A: rank + dim(null space) = n.if dim(null space)= 0 then A is full rank.In this case, the action of the matrix is invertible.The inversion is also linear and consequently can be represented byanother matrix A−1.A−1 is the only matrix such that A−1A = AA−1 = I .
32 / 50
![Page 70: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/70.jpg)
Matrix Inverse
Something NotableFor an n × n matrix A: rank + dim(null space) = n.if dim(null space)= 0 then A is full rank.In this case, the action of the matrix is invertible.The inversion is also linear and consequently can be represented byanother matrix A−1.A−1 is the only matrix such that A−1A = AA−1 = I .
32 / 50
![Page 71: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/71.jpg)
Matrix Inverse
Something NotableFor an n × n matrix A: rank + dim(null space) = n.if dim(null space)= 0 then A is full rank.In this case, the action of the matrix is invertible.The inversion is also linear and consequently can be represented byanother matrix A−1.A−1 is the only matrix such that A−1A = AA−1 = I .
32 / 50
![Page 72: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/72.jpg)
Matrix Inverse
Something NotableFor an n × n matrix A: rank + dim(null space) = n.if dim(null space)= 0 then A is full rank.In this case, the action of the matrix is invertible.The inversion is also linear and consequently can be represented byanother matrix A−1.A−1 is the only matrix such that A−1A = AA−1 = I .
32 / 50
![Page 73: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/73.jpg)
Orthogonal Matrices
DefinitionAn orthogonal matrix U satisfies U T U = I .
PropertiesU has orthonormal columns.
In additionApplying an orthogonal matrix to two vectors does not change their innerproduct: 〈Uu,Uv〉 = (Uu)T Uv
=uT U T Uv=uT v= 〈u, v〉
33 / 50
![Page 74: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/74.jpg)
Orthogonal Matrices
DefinitionAn orthogonal matrix U satisfies U T U = I .
PropertiesU has orthonormal columns.
In additionApplying an orthogonal matrix to two vectors does not change their innerproduct: 〈Uu,Uv〉 = (Uu)T Uv
=uT U T Uv=uT v= 〈u, v〉
33 / 50
![Page 75: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/75.jpg)
Orthogonal Matrices
DefinitionAn orthogonal matrix U satisfies U T U = I .
PropertiesU has orthonormal columns.
In additionApplying an orthogonal matrix to two vectors does not change their innerproduct: 〈Uu,Uv〉 = (Uu)T Uv
=uT U T Uv=uT v= 〈u, v〉
33 / 50
![Page 76: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/76.jpg)
Example
A classic oneMatrices representing rotations are orthogonal.
34 / 50
![Page 77: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/77.jpg)
Outline
1 IntroductionWhat is a Vector?
2 Vector SpacesDefinitionLinear Independence and Basis of Vector SpacesNorm of a VectorInner ProductMatricesTrace and DeterminantMatrix DecompositionSingular Value Decomposition
35 / 50
![Page 78: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/78.jpg)
Trace and Determinant
Definition (Trace)The trace is the sum of the diagonal elements of a square matrix.
Definition (Determinant)The determinant of a square matrix A, denoted by |A|, is defined as
det (A) =n∑
j=1(−1)i+j aijMij
where Mij is determinant of matrix A without the row i and column j.
36 / 50
![Page 79: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/79.jpg)
Trace and Determinant
Definition (Trace)The trace is the sum of the diagonal elements of a square matrix.
Definition (Determinant)The determinant of a square matrix A, denoted by |A|, is defined as
det (A) =n∑
j=1(−1)i+j aijMij
where Mij is determinant of matrix A without the row i and column j.
36 / 50
![Page 80: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/80.jpg)
Special Case
For a 2× 2 matrix A =(
a bc d
)
|A| = ad − bc
The absolute value of |A|is the area of the parallelogram given by therows of A
37 / 50
![Page 81: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/81.jpg)
Special Case
For a 2× 2 matrix A =(
a bc d
)
|A| = ad − bc
The absolute value of |A|is the area of the parallelogram given by therows of A
37 / 50
![Page 82: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/82.jpg)
Properties of the Determinant
Basic Properties|A| =
∣∣∣AT∣∣∣
|AB| = |A| |B||A| = 0 if and only if A is not invertibleIf A is invertible, then
∣∣A−1∣∣ = 1|A| .
38 / 50
![Page 83: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/83.jpg)
Properties of the Determinant
Basic Properties|A| =
∣∣∣AT∣∣∣
|AB| = |A| |B||A| = 0 if and only if A is not invertibleIf A is invertible, then
∣∣A−1∣∣ = 1|A| .
38 / 50
![Page 84: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/84.jpg)
Properties of the Determinant
Basic Properties|A| =
∣∣∣AT∣∣∣
|AB| = |A| |B||A| = 0 if and only if A is not invertibleIf A is invertible, then
∣∣A−1∣∣ = 1|A| .
38 / 50
![Page 85: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/85.jpg)
Properties of the Determinant
Basic Properties|A| =
∣∣∣AT∣∣∣
|AB| = |A| |B||A| = 0 if and only if A is not invertibleIf A is invertible, then
∣∣A−1∣∣ = 1|A| .
38 / 50
![Page 86: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/86.jpg)
Outline
1 IntroductionWhat is a Vector?
2 Vector SpacesDefinitionLinear Independence and Basis of Vector SpacesNorm of a VectorInner ProductMatricesTrace and DeterminantMatrix DecompositionSingular Value Decomposition
39 / 50
![Page 87: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/87.jpg)
Eigenvalues and Eigenvectors
EigenvaluesAn eigenvalue λ of a square matrix A satisfies:
Au = λu
for some vector , which we call an eigenvector.
PropertiesGeometrically the operator A expands when (λ > 1) or contracts (λ < 1)eigenvectors, but does not rotate them.
Null Space relationIf u is an eigenvector of A, it is in the null space of A− λI , which isconsequently not invertible.
40 / 50
![Page 88: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/88.jpg)
Eigenvalues and Eigenvectors
EigenvaluesAn eigenvalue λ of a square matrix A satisfies:
Au = λu
for some vector , which we call an eigenvector.
PropertiesGeometrically the operator A expands when (λ > 1) or contracts (λ < 1)eigenvectors, but does not rotate them.
Null Space relationIf u is an eigenvector of A, it is in the null space of A− λI , which isconsequently not invertible.
40 / 50
![Page 89: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/89.jpg)
Eigenvalues and Eigenvectors
EigenvaluesAn eigenvalue λ of a square matrix A satisfies:
Au = λu
for some vector , which we call an eigenvector.
PropertiesGeometrically the operator A expands when (λ > 1) or contracts (λ < 1)eigenvectors, but does not rotate them.
Null Space relationIf u is an eigenvector of A, it is in the null space of A− λI , which isconsequently not invertible.
40 / 50
![Page 90: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/90.jpg)
More properties
Given the previous relationThe eigenvalues of A are the roots of the equation |A− λI | = 0
Remark: We do not calculate the eigenvalues this way
Something NotableEigenvalues and eigenvectors can be complex valued, even if all the entriesof A are real.
41 / 50
![Page 91: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/91.jpg)
More properties
Given the previous relationThe eigenvalues of A are the roots of the equation |A− λI | = 0
Remark: We do not calculate the eigenvalues this way
Something NotableEigenvalues and eigenvectors can be complex valued, even if all the entriesof A are real.
41 / 50
![Page 92: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/92.jpg)
Eigendecomposition of a Matrix
GivenLet A be an n × n square matrix with n linearly independent eigenvectorsp1,p2, ...,pn and eigenvalues λ1, λ2, ..., λn
We define the matrices
P = (p1 p2 · · ·pn)
Λ =
λ1 0 · · · 00 λ2 · · · 0
· · ·0 0 · · · λn
42 / 50
![Page 93: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/93.jpg)
Eigendecomposition of a Matrix
GivenLet A be an n × n square matrix with n linearly independent eigenvectorsp1,p2, ...,pn and eigenvalues λ1, λ2, ..., λn
We define the matrices
P = (p1 p2 · · ·pn)
Λ =
λ1 0 · · · 00 λ2 · · · 0
· · ·0 0 · · · λn
42 / 50
![Page 94: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/94.jpg)
Properties
We have that A satisfies
AP = PΛ
In additionP is full rank.
Thus, inverting it yields the eigendecomposition
A = PΛP−1
43 / 50
![Page 95: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/95.jpg)
Properties
We have that A satisfies
AP = PΛ
In additionP is full rank.
Thus, inverting it yields the eigendecomposition
A = PΛP−1
43 / 50
![Page 96: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/96.jpg)
Properties
We have that A satisfies
AP = PΛ
In additionP is full rank.
Thus, inverting it yields the eigendecomposition
A = PΛP−1
43 / 50
![Page 97: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/97.jpg)
Properties of the Eigendecomposition
We have thatNot all matrices are diagonalizable/eigendecomposition. Example(
1 10 1
)Trace (A) = Trace (Λ) =
∑ni=1 λi
|A| = |Λ| = Πni=1λi
The rank of A is equal to the number of nonzero eigenvalues.If λ is anonzero eigenvalue of A, 1
λ is an eigenvalue of A−1 with thesame eigenvector.The eigendecompositon allows to compute matrix powers efficiently:
Am =(PΛP−1)m = PΛP−1PΛP−1PΛP−1 . . .PΛP−1 =
PΛmP−1
44 / 50
![Page 98: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/98.jpg)
Properties of the Eigendecomposition
We have thatNot all matrices are diagonalizable/eigendecomposition. Example(
1 10 1
)Trace (A) = Trace (Λ) =
∑ni=1 λi
|A| = |Λ| = Πni=1λi
The rank of A is equal to the number of nonzero eigenvalues.If λ is anonzero eigenvalue of A, 1
λ is an eigenvalue of A−1 with thesame eigenvector.The eigendecompositon allows to compute matrix powers efficiently:
Am =(PΛP−1)m = PΛP−1PΛP−1PΛP−1 . . .PΛP−1 =
PΛmP−1
44 / 50
![Page 99: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/99.jpg)
Properties of the Eigendecomposition
We have thatNot all matrices are diagonalizable/eigendecomposition. Example(
1 10 1
)Trace (A) = Trace (Λ) =
∑ni=1 λi
|A| = |Λ| = Πni=1λi
The rank of A is equal to the number of nonzero eigenvalues.If λ is anonzero eigenvalue of A, 1
λ is an eigenvalue of A−1 with thesame eigenvector.The eigendecompositon allows to compute matrix powers efficiently:
Am =(PΛP−1)m = PΛP−1PΛP−1PΛP−1 . . .PΛP−1 =
PΛmP−1
44 / 50
![Page 100: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/100.jpg)
Properties of the Eigendecomposition
We have thatNot all matrices are diagonalizable/eigendecomposition. Example(
1 10 1
)Trace (A) = Trace (Λ) =
∑ni=1 λi
|A| = |Λ| = Πni=1λi
The rank of A is equal to the number of nonzero eigenvalues.If λ is anonzero eigenvalue of A, 1
λ is an eigenvalue of A−1 with thesame eigenvector.The eigendecompositon allows to compute matrix powers efficiently:
Am =(PΛP−1)m = PΛP−1PΛP−1PΛP−1 . . .PΛP−1 =
PΛmP−1
44 / 50
![Page 101: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/101.jpg)
Properties of the Eigendecomposition
We have thatNot all matrices are diagonalizable/eigendecomposition. Example(
1 10 1
)Trace (A) = Trace (Λ) =
∑ni=1 λi
|A| = |Λ| = Πni=1λi
The rank of A is equal to the number of nonzero eigenvalues.If λ is anonzero eigenvalue of A, 1
λ is an eigenvalue of A−1 with thesame eigenvector.The eigendecompositon allows to compute matrix powers efficiently:
Am =(PΛP−1)m = PΛP−1PΛP−1PΛP−1 . . .PΛP−1 =
PΛmP−1
44 / 50
![Page 102: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/102.jpg)
Properties of the Eigendecomposition
We have thatNot all matrices are diagonalizable/eigendecomposition. Example(
1 10 1
)Trace (A) = Trace (Λ) =
∑ni=1 λi
|A| = |Λ| = Πni=1λi
The rank of A is equal to the number of nonzero eigenvalues.If λ is anonzero eigenvalue of A, 1
λ is an eigenvalue of A−1 with thesame eigenvector.The eigendecompositon allows to compute matrix powers efficiently:
Am =(PΛP−1)m = PΛP−1PΛP−1PΛP−1 . . .PΛP−1 =
PΛmP−1
44 / 50
![Page 103: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/103.jpg)
Eigendecomposition of a Symmetric Matrix
When A symmetric, we haveIf A = AT then A is symmetric.The eigenvalues of symmetric matrices are real.The eigenvectors of symmetric matrices are orthonormal.Consequently, the eigendecomposition becomes A = UΛU T for Λreal and U orthogonal.The eigenvectors of A are an orthonormal basis for the column spaceand row space.
45 / 50
![Page 104: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/104.jpg)
Eigendecomposition of a Symmetric Matrix
When A symmetric, we haveIf A = AT then A is symmetric.The eigenvalues of symmetric matrices are real.The eigenvectors of symmetric matrices are orthonormal.Consequently, the eigendecomposition becomes A = UΛU T for Λreal and U orthogonal.The eigenvectors of A are an orthonormal basis for the column spaceand row space.
45 / 50
![Page 105: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/105.jpg)
Eigendecomposition of a Symmetric Matrix
When A symmetric, we haveIf A = AT then A is symmetric.The eigenvalues of symmetric matrices are real.The eigenvectors of symmetric matrices are orthonormal.Consequently, the eigendecomposition becomes A = UΛU T for Λreal and U orthogonal.The eigenvectors of A are an orthonormal basis for the column spaceand row space.
45 / 50
![Page 106: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/106.jpg)
Eigendecomposition of a Symmetric Matrix
When A symmetric, we haveIf A = AT then A is symmetric.The eigenvalues of symmetric matrices are real.The eigenvectors of symmetric matrices are orthonormal.Consequently, the eigendecomposition becomes A = UΛU T for Λreal and U orthogonal.The eigenvectors of A are an orthonormal basis for the column spaceand row space.
45 / 50
![Page 107: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/107.jpg)
Eigendecomposition of a Symmetric Matrix
When A symmetric, we haveIf A = AT then A is symmetric.The eigenvalues of symmetric matrices are real.The eigenvectors of symmetric matrices are orthonormal.Consequently, the eigendecomposition becomes A = UΛU T for Λreal and U orthogonal.The eigenvectors of A are an orthonormal basis for the column spaceand row space.
45 / 50
![Page 108: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/108.jpg)
We can see the action of a symmetric matrix on a vector uas...
We can decompose the action Au = UΛU Tu asProjection of u onto the column space of A (Multiplication by U T ).Scaling of each coefficient 〈U i ,u〉 by the corresponding eigenvalue(Multiplication by Λ).Linear combination of the eigenvectors scaled by the resultingcoefficient (Multiplication by U ).
Final equation
Au =n∑
i=1λi 〈U i ,u〉U i
It would be great to generalize this to all matrices!!!
46 / 50
![Page 109: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/109.jpg)
We can see the action of a symmetric matrix on a vector uas...
We can decompose the action Au = UΛU Tu asProjection of u onto the column space of A (Multiplication by U T ).Scaling of each coefficient 〈U i ,u〉 by the corresponding eigenvalue(Multiplication by Λ).Linear combination of the eigenvectors scaled by the resultingcoefficient (Multiplication by U ).
Final equation
Au =n∑
i=1λi 〈U i ,u〉U i
It would be great to generalize this to all matrices!!!
46 / 50
![Page 110: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/110.jpg)
We can see the action of a symmetric matrix on a vector uas...
We can decompose the action Au = UΛU Tu asProjection of u onto the column space of A (Multiplication by U T ).Scaling of each coefficient 〈U i ,u〉 by the corresponding eigenvalue(Multiplication by Λ).Linear combination of the eigenvectors scaled by the resultingcoefficient (Multiplication by U ).
Final equation
Au =n∑
i=1λi 〈U i ,u〉U i
It would be great to generalize this to all matrices!!!
46 / 50
![Page 111: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/111.jpg)
We can see the action of a symmetric matrix on a vector uas...
We can decompose the action Au = UΛU Tu asProjection of u onto the column space of A (Multiplication by U T ).Scaling of each coefficient 〈U i ,u〉 by the corresponding eigenvalue(Multiplication by Λ).Linear combination of the eigenvectors scaled by the resultingcoefficient (Multiplication by U ).
Final equation
Au =n∑
i=1λi 〈U i ,u〉U i
It would be great to generalize this to all matrices!!!
46 / 50
![Page 112: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/112.jpg)
We can see the action of a symmetric matrix on a vector uas...
We can decompose the action Au = UΛU Tu asProjection of u onto the column space of A (Multiplication by U T ).Scaling of each coefficient 〈U i ,u〉 by the corresponding eigenvalue(Multiplication by Λ).Linear combination of the eigenvectors scaled by the resultingcoefficient (Multiplication by U ).
Final equation
Au =n∑
i=1λi 〈U i ,u〉U i
It would be great to generalize this to all matrices!!!
46 / 50
![Page 113: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/113.jpg)
Outline
1 IntroductionWhat is a Vector?
2 Vector SpacesDefinitionLinear Independence and Basis of Vector SpacesNorm of a VectorInner ProductMatricesTrace and DeterminantMatrix DecompositionSingular Value Decomposition
47 / 50
![Page 114: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/114.jpg)
Singular Value Decomposition
Every Matrix has a singular value decomposition
A = UΣV T
WhereThe columns of U are an orthonormal basis for the column space.The columns of V are an orthonormal basis for the row space.The Σ is diagonal and the entries on its diagonal σi = Σii are positivereal numbers, called the singular values of A.
The action of Aon a vector u can be decomposed intoAu =
∑ni=1 σi 〈V i ,u〉U i
48 / 50
![Page 115: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/115.jpg)
Singular Value Decomposition
Every Matrix has a singular value decomposition
A = UΣV T
WhereThe columns of U are an orthonormal basis for the column space.The columns of V are an orthonormal basis for the row space.The Σ is diagonal and the entries on its diagonal σi = Σii are positivereal numbers, called the singular values of A.
The action of Aon a vector u can be decomposed intoAu =
∑ni=1 σi 〈V i ,u〉U i
48 / 50
![Page 116: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/116.jpg)
Singular Value Decomposition
Every Matrix has a singular value decomposition
A = UΣV T
WhereThe columns of U are an orthonormal basis for the column space.The columns of V are an orthonormal basis for the row space.The Σ is diagonal and the entries on its diagonal σi = Σii are positivereal numbers, called the singular values of A.
The action of Aon a vector u can be decomposed intoAu =
∑ni=1 σi 〈V i ,u〉U i
48 / 50
![Page 117: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/117.jpg)
Singular Value Decomposition
Every Matrix has a singular value decomposition
A = UΣV T
WhereThe columns of U are an orthonormal basis for the column space.The columns of V are an orthonormal basis for the row space.The Σ is diagonal and the entries on its diagonal σi = Σii are positivereal numbers, called the singular values of A.
The action of Aon a vector u can be decomposed intoAu =
∑ni=1 σi 〈V i ,u〉U i
48 / 50
![Page 118: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/118.jpg)
Properties of the Singular Value Decomposition
FirstThe eigenvalues of the symmetric matrix AT A are equal to the square ofthe singular values of A:
AT A = V ΣU T U T ΣV T = V Σ2V T
SecondThe rank of a matrix is equal to the number of nonzero singular values.
ThirdThe largest singular value σ1 is the solution to the optimization problem:
σ1 = maxx 6=0
‖Ax‖2‖x‖2
49 / 50
![Page 119: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/119.jpg)
Properties of the Singular Value Decomposition
FirstThe eigenvalues of the symmetric matrix AT A are equal to the square ofthe singular values of A:
AT A = V ΣU T U T ΣV T = V Σ2V T
SecondThe rank of a matrix is equal to the number of nonzero singular values.
ThirdThe largest singular value σ1 is the solution to the optimization problem:
σ1 = maxx 6=0
‖Ax‖2‖x‖2
49 / 50
![Page 120: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/120.jpg)
Properties of the Singular Value Decomposition
FirstThe eigenvalues of the symmetric matrix AT A are equal to the square ofthe singular values of A:
AT A = V ΣU T U T ΣV T = V Σ2V T
SecondThe rank of a matrix is equal to the number of nonzero singular values.
ThirdThe largest singular value σ1 is the solution to the optimization problem:
σ1 = maxx 6=0
‖Ax‖2‖x‖2
49 / 50
![Page 121: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/121.jpg)
Properties of the Singular Value Decomposition
RemarkIt can be verified that the largest singular value satisfies the properties of anorm, it is called the spectral norm of the matrix.
FinallyIn statistics analyzing data with the singular value decomposition is calledPrincipal Component Analysis.
50 / 50
![Page 122: 03 Machine Learning Linear Algebra](https://reader034.vdocument.in/reader034/viewer/2022051404/586e84ef1a28aba0038b61fd/html5/thumbnails/122.jpg)
Properties of the Singular Value Decomposition
RemarkIt can be verified that the largest singular value satisfies the properties of anorm, it is called the spectral norm of the matrix.
FinallyIn statistics analyzing data with the singular value decomposition is calledPrincipal Component Analysis.
50 / 50