linear algebra primer - artificial...
TRANSCRIPT
![Page 1: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/1.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
1
Linear Algebra Primer
Juan Carlos Niebles and Ranjay KrishnaStanford Vision and Learning Lab
Another, very in-depth linear algebra review from CS229 is available here:http://cs229.stanford.edu/section/cs229-linalg.pdfAnd a video discussion of linear algebra from EE263 is here (lectures 3 and 4):https://see.stanford.edu/Course/EE263
![Page 2: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/2.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
2
Outline• Vectors and matrices– Basic Matrix Operations– Determinants, norms, trace– Special Matrices
• Transformation Matrices– Homogeneous coordinates– Translation
• Matrix inverse• Matrix rank• Eigenvalues and Eigenvectors• Matrix Calculus
![Page 3: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/3.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
3
Outline• Vectors and matrices– Basic Matrix Operations– Determinants, norms, trace– Special Matrices
• Transformation Matrices– Homogeneous coordinates– Translation
• Matrix inverse• Matrix rank• Eigenvalues and Eigenvectors• Matrix Calculus
Vectors and matrices are just collections of ordered numbers that represent something: movements in space, scaling factors, pixel brightness, etc. We’ll define some common uses and standard operations on them.
![Page 4: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/4.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
4
Vector• A column vector where
• A row vector where
denotes the transpose operation
![Page 5: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/5.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
5
Vector• We’ll default to column vectors in this class
• You’ll want to keep track of the orientation of your vectors when programming in python
• You can transpose a vector V in python by writing V.t. (But in class materials, we will always use VT to indicate transpose, and we will use V’ to mean “V prime”)
![Page 6: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/6.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
6
Vectors have two main uses
• Vectors can represent an offset in 2D or 3D space.
• Points are just vectors from the origin.
• Data (pixels, gradients at an image keypoint, etc) can also be treated as a vector.
• Such vectors don’t have a geometric interpretation, but calculations like “distance” can still have value.
![Page 7: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/7.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
7
Matrix• A matrix is an array of numbers with size by ,
i.e. m rows and n columns.
• If , we say that is square.
7
![Page 8: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/8.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
8
Images
8
• Python represents an image as a matrix of pixel brightnesses
• Note that the upper left corner is [y,x] = (0,0)
=
![Page 9: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/9.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
9
Images as both a matrix as well as a vector
![Page 10: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/10.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
10
Color Images• Grayscale images have one number per pixel, and are
stored as an m × n matrix.• Color images have 3 numbers per pixel – red, green,
and blue brightnesses (RGB)• Stored as an m × n × 3 matrix
=
![Page 11: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/11.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
11
Basic Matrix Operations• We will discuss:– Addition– Scaling– Dot product–Multiplication– Transpose– Inverse / pseudoinverse– Determinant / trace
![Page 12: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/12.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
12
Matrix Operations• Addition
– Can only add a matrix with matching dimensions, or a scalar.
• Scaling
![Page 13: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/13.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
13
•Norm• More formally, a norm is any function
that satisfies 4 properties:
• Non-negativity: For all • Definiteness: f(x) = 0 if and only if x = 0. • Homogeneity: For all • Triangle inequality: For all
Vectors
![Page 14: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/14.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
14
• Example Norms
• General norms:
Matrix Operations
![Page 15: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/15.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
15
Matrix Operations• Inner product (dot product) of vectors–Multiply corresponding entries of two vectors and
add up the result– x·y is also |x||y|Cos( the angle between x and y )
![Page 16: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/16.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
16
Matrix Operations• Inner product (dot product) of vectors– If B is a unit vector, then A·B gives the length of A
which lies in the direction of B
![Page 17: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/17.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
17
Matrix Operations• The product of two matrices
![Page 18: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/18.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
18
Matrix Operations• Multiplication
• The product AB is:
• Each entry in the result is (that row of A) dot product with (that column of B)
• Many uses, which will be covered later
![Page 19: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/19.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
19
Matrix Operations• Multiplication example:
– Each entry of the matrix product is made by taking the dot product of the corresponding row in the left matrix, with the corresponding column in the right one.
![Page 20: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/20.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
20
Matrix Operations• The product of two matrices
![Page 21: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/21.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
21
Matrix Operations
• Powers– By convention, we can refer to the matrix product
AA as A2, and AAA as A3, etc.– Obviously only square matrices can be multiplied
that way
![Page 22: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/22.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
22
Matrix Operations• Transpose – flip matrix, so row 1 becomes
column 1
• A useful identity:
![Page 23: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/23.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
23
• Determinant– returns a scalar– Represents area (or volume) of the
parallelogram described by the vectors in the rows of the matrix
– For , – Properties:
Matrix Operations
![Page 24: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/24.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
24
• Trace
– Invariant to a lot of transformations, so it’s used sometimes in proofs. (Rarely in this class though.)
– Properties:
Matrix Operations
![Page 25: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/25.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
25
• Vector Norms
• Matrix norms: Norms can also be defined for matrices, such as the Frobenius norm:
Matrix Operations
![Page 26: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/26.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
26
Special Matrices• Identity matrix I– Square matrix, 1’s along diagonal, 0’s
elsewhere– I · [another matrix] = [that matrix]
• Diagonal matrix– Square matrix with numbers along
diagonal, 0’s elsewhere– A diagonal · [another matrix] scales the
rows of that matrix
![Page 27: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/27.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
27
Special Matrices• Symmetric matrix
• Skew-symmetric matrix 2
40 �2 �52 0 �75 7 0
3
5
![Page 28: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/28.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
28
Outline• Vectors and matrices– Basic Matrix Operations– Determinants, norms, trace– Special Matrices
• Transformation Matrices– Homogeneous coordinates– Translation
• Matrix inverse• Matrix rank• Eigenvalues and Eigenvectors• Matrix Calculus
Matrix multiplication can be used to transform vectors. A matrix used in this way is called a transformation matrix.
![Page 29: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/29.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
29
Transformation• Matrices can be used to transform vectors in useful ways,
through multiplication: x’= Ax• Simplest is scaling:
(Verify to yourself that the matrix multiplication works out this way)
![Page 30: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/30.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
30
Transformation
![Page 31: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/31.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
31
Rotation
! = 45°
![Page 32: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/32.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
32
Rotation• How can you convert a vector represented in frame
“0” to a new, rotated coordinate frame “1”?
![Page 33: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/33.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
33
Rotation• How can you convert a vector represented in frame
“0” to a new, rotated coordinate frame “1”?• Remember what a vector is:
[component in direction of the frame’s x axis, component in direction of y axis]
![Page 34: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/34.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
34
Rotation• So to rotate it we must produce this vector:
[component in direction of new x axis, component in direction of new y axis]• We can do this easily with dot products!• New x coordinate is [original vector] dot [the new x axis]• New y coordinate is [original vector] dot [the new y axis]
![Page 35: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/35.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
35
Rotation• Insight: this is what happens in a matrix*vector
multiplication– Result x coordinate is:
[original vector] dot [matrix row 1]– So matrix multiplication can rotate a vector p:
![Page 36: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/36.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
36
Rotation• Suppose we express a point in the new
coordinate system which is rotated left• If we plot the result in the original coordinate
system, we have rotated the point right– Thus, rotation matrices
can be used to rotate vectors. We’ll usually think of them in that sense-- as operators to rotate vectors
![Page 37: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/37.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
37
2D Rotation Matrix FormulaCounter-clockwise rotation by an angle q
úû
ùêë
éúû
ùêë
é -=ú
û
ùêë
éyx
yx
qqqq
cossinsincos
''
P
x
y’P’
q
x’
y
PRP'=
yθsinxθcos'x -=xθsinyθcos'y +=
![Page 38: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/38.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
38
Transformation Matrices• Multiple transformation matrices can be used to transform a
point: p’=R2 R1 S p
• The effect of this is to apply their transformations one after the other, from right to left.
• In the example above, the result is (R2 (R1 (S p)))
• The result is exactly the same if we multiply the matrices first, to form a single transformation matrix:p’=(R2 R1 S) p
![Page 39: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/39.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
39
Homogeneous system
• In general, a matrix multiplication lets us linearly combine components of a vector
– This is sufficient for scale, rotate, skew transformations.– But notice, we can’t add a constant! L
![Page 40: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/40.jpg)
Linear Algebra Review
Stanford University
27-Se
p-20
18
40
Homogeneous system
– The (somewhat hacky) solution? Stick a “1” at the end of every vector:
– Now we can rotate, scale, and skew like before, AND translate(note how the multiplication works out, above)
– This is called “homogeneous coordinates”
![Page 41: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/41.jpg)
Linear Algebra Review
Stanford University
27-Se
p-20
18
41
Homogeneous system– In homogeneous coordinates, the multiplication works out
so the rightmost column of the matrix is a vector that gets added.
– Generally, a homogeneous transformation matrix will have a bottom row of [0 0 1], so that the result has a “1” at the bottom too.
![Page 42: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/42.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
42
Homogeneous system• One more thing we might want: to divide the result by
something– For example, we may want to divide by a coordinate, to make
things scale down as they get farther away in a camera image–Matrix multiplication can’t actually divide– So, by convention, in homogeneous coordinates, we’ll divide the
result by its last coordinate after doing a matrix multiplication
![Page 43: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/43.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
43
2D Translation
t
P
P’
![Page 44: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/44.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
44
P
x
y
tx
ty
P’t
úúú
û
ù
êêê
ë
é×úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é++
®1100
1001
1' y
xtt
tytx
y
x
y
x
P
)1,,(),()1,,(),(
yxyx ttttyxyx
®=®=
tP
P
2D Translation using Homogeneous Coordinates
![Page 45: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/45.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
45
P
x
y
tx
ty
P’t
úúú
û
ù
êêê
ë
é×úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é++
®1100
1001
1' y
xtt
tytx
y
x
y
x
P
)1,,(),()1,,(),(
yxyx ttttyxyx
®=®=
tP
P
2D Translation using Homogeneous Coordinates
![Page 46: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/46.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
46
P
x
y
tx
ty
P’t
úúú
û
ù
êêê
ë
é×úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é++
®1100
1001
1' y
xtt
tytx
y
x
y
x
P
)1,,(),()1,,(),(
yxyx ttttyxyx
®=®=
tP
P
2D Translation using Homogeneous Coordinates
![Page 47: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/47.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
47
P
x
y
tx
ty
P’t
úúú
û
ù
êêê
ë
é×úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é++
®1100
1001
1' y
xtt
tytx
y
x
y
x
P
)1,,(),()1,,(),(
yxyx ttttyxyx
®=®=
tP
P
2D Translation using Homogeneous Coordinates
![Page 48: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/48.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
48
P
x
y
tx
ty
P’t
úúú
û
ù
êêê
ë
é×úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é++
®1100
1001
1' y
xtt
tytx
y
x
y
x
P
)1,,(),()1,,(),(
yxyx ttttyxyx
®=®=
tP
tP
PTPtI
×=×úû
ùêë
é=
10
2D Translation using Homogeneous Coordinates
![Page 49: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/49.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
49
Scaling
P
P’
![Page 50: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/50.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
50
Scaling Equation
P
x
y
sx x
P’sy y
)1,,(),(')1,,(),(
ysxsysxsyxyx
yxyx ®=®=
PP
)ys,xs(')y,x( yx=®= PP
![Page 51: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/51.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
51
Scaling Equation
P
x
y
sx x
P’sy y
úúú
û
ù
êêê
ë
é×úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é®
11000000
1' y
xs
sysxs
y
x
y
x
P
)1,,(),(')1,,(),(
ysxsysxsyxyx
yxyx ®=®=
PP
)ys,xs(')y,x( yx=®= PP
![Page 52: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/52.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
52
Scaling Equation
P
x
y
sx x
P’sy y
úúú
û
ù
êêê
ë
é×úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é®
11000000
1' y
xs
sysxs
y
x
y
x
P
)1,,(),(')1,,(),(
ysxsysxsyxyx
yxyx ®=®=
PP
S
PSP100S
×=×úû
ùêë
é=
'
)ys,xs(')y,x( yx=®= PP
![Page 53: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/53.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
53
P
P’=S∙PP’’=T∙P’
P’’=T · P’=T ·(S · P)= T · S ·P
Scaling & Translating
P’’
![Page 54: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/54.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
54
Scaling & Translating
P '' = T ⋅S ⋅P =1 0 tx0 1 ty0 0 1
"
#
$$$$
%
&
''''
⋅
sx 0 00 sy 0
0 0 1
"
#
$$$$
%
&
''''
xy1
"
#
$$$
%
&
'''=
=
sx 0 tx0 sy ty0 0 1
"
#
$$$$
%
&
''''
xy1
"
#
$$$
%
&
'''=
sxx + txsyy+ ty1
"
#
$$$$
%
&
''''
= S t0 1
"
#$
%
&'
xy1
"
#
$$$
%
&
'''
A
![Page 55: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/55.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
55
Scaling & Translating
P '' = T ⋅S ⋅P =1 0 tx0 1 ty0 0 1
"
#
$$$$
%
&
''''
⋅
sx 0 00 sy 0
0 0 1
"
#
$$$$
%
&
''''
xy1
"
#
$$$
%
&
'''=
=
sx 0 tx0 sy ty0 0 1
"
#
$$$$
%
&
''''
xy1
"
#
$$$
%
&
'''=
sxx + txsyy+ ty1
"
#
$$$$
%
&
''''
= S t0 1
"
#$
%
&'
xy1
"
#
$$$
%
&
'''
![Page 56: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/56.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
56
úúú
û
ù
êêê
ë
é++
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=××=
1tystxs
1yx
100ts0t0s
1yx
1000s000s
100t10t01
''' yy
xx
yy
xx
y
x
y
x
PSTP
Translating & Scaling != Scaling & Translating
![Page 57: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/57.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
57
úúú
û
ù
êêê
ë
é++
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=××=
1tsystsxs
1yx
100tss0ts0s
1yx
100t10t01
1000s000s
'''
yyy
xxx
yyy
xxx
y
x
y
x
PTSP
úúú
û
ù
êêê
ë
é++
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=××=
1tystxs
1yx
100ts0t0s
1yx
1000s000s
100t10t01
''' yy
xx
yy
xx
y
x
y
x
PSTP
Translating & Scaling != Scaling & Translating
![Page 58: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/58.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
58
Translating & Scaling != Scaling & Translating
úúú
û
ù
êêê
ë
é++
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=××=
1tsystsxs
1yx
100tss0ts0s
1yx
100t10t01
1000s000s
'''
yyy
xxx
yyy
xxx
y
x
y
x
PTSP
úúú
û
ù
êêê
ë
é++
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=××=
1tystxs
1yx
100ts0t0s
1yx
1000s000s
100t10t01
''' yy
xx
yy
xx
y
x
y
x
PSTP
![Page 59: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/59.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
59
Rotation
P
P’
![Page 60: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/60.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
60
Rotation EquationsCounter-clockwise rotation by an angle q
úû
ùêë
éúû
ùêë
é -=ú
û
ùêë
éyx
yx
qqqq
cossinsincos
''
P
x
y’P’
q
x’
y
PRP'=
yθsinxθcos'x -=xθsinyθcos'y +=
![Page 61: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/61.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
61
Rotation Matrix Properties
A 2D rotation matrix is 2x2
1)det( ==×=×
RIRRRR TT
úû
ùêë
éúû
ùêë
é -=ú
û
ùêë
éyx
yx
qqqq
cossinsincos
''
Note: R belongs to the category of normal matrices and satisfies many interesting properties:
![Page 62: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/62.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
62
Rotation Matrix Properties• Transpose of a rotation matrix produces a
rotation in the opposite direction
• The rows of a rotation matrix are always mutually perpendicular (a.k.a. orthogonal) unit vectors– (and so are its columns)
1)det( ==×=×
RIRRRR TT
![Page 63: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/63.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
63
Scaling + Rotation + TranslationP’= (T R S) P
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é -
úúú
û
ù
êêê
ë
é=×××=
1yx
1000s000s
1000θcossinθ0inθsθcos
100t10t01
R' y
x
y
x
PSTP
=úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é -=
1yx
1000s000s
100tθcossinθtinθsθcos
y
x
y
x
úúú
û
ù
êêê
ë
é
úû
ùêë
é=
úúú
û
ù
êêê
ë
é
úû
ùêë
éúû
ùêë
é=
110
1100
10yx
tSRyx
StR
This is the form of the general-purpose transformation matrix
![Page 64: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/64.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
64
Outline• Vectors and matrices– Basic Matrix Operations– Determinants, norms, trace– Special Matrices
• Transformation Matrices– Homogeneous coordinates– Translation
• Matrix inverse• Matrix rank• Eigenvalues and Eigenvectors• Matrix Calculate
The inverse of a transformation matrix reverses its effect
![Page 65: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/65.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
65
• Given a matrix A, its inverse A-1 is a matrix such that AA-1 = A-1A = I
• E.g.
• Inverse does not always exist. If A-1 exists, A is invertible or non-singular. Otherwise, it’s singular.
• Useful identities, for matrices that are invertible:
Inverse
![Page 66: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/66.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
66
• Pseudoinverse– Fortunately, there are workarounds to solve AX=B in these
situations. And python can do them!– Instead of taking an inverse, directly ask python to solve for X in
AX=B, by typing np.linalg.solve(A, B)– Python will try several appropriate numerical methods (including
the pseudoinverse if the inverse doesn’t exist)– Python will return the value of X which solves the equation• If there is no exact solution, it will return the closest one• If there are many solutions, it will return the smallest one
Matrix Operations
![Page 67: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/67.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
67
• Python example:
Matrix Operations
>> import numpy as np>> x = np.linalg.solve(A,B)x =
1.0000-0.5000
![Page 68: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/68.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
68
Outline• Vectors and matrices– Basic Matrix Operations– Determinants, norms, trace– Special Matrices
• Transformation Matrices– Homogeneous coordinates– Translation
• Matrix inverse• Matrix rank• Eigenvalues and Eigenvectors• Matrix Calculate
The rank of a transformation matrix tells you how many dimensions it transforms a vector to.
![Page 69: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/69.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
69
Linear independence• Suppose we have a set of vectors v1, …, vn• If we can express v1 as a linear combination of the other
vectors v2…vn, then v1 is linearly dependent on the other vectors. – The direction v1 can be expressed as a combination of the
directions v2…vn. (E.g. v1 = .7 v2 -.7 v4)
![Page 70: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/70.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
70
Linear independence• Suppose we have a set of vectors v1, …, vn• If we can express v1 as a linear combination of the other
vectors v2…vn, then v1 is linearly dependent on the other vectors. – The direction v1 can be expressed as a combination of the
directions v2…vn. (E.g. v1 = .7 v2 -.7 v4)
• If no vector is linearly dependent on the rest of the set, the set is linearly independent.– Common case: a set of vectors v1, …, vn is always linearly
independent if each vector is perpendicular to every other vector (and non-zero)
![Page 71: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/71.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
71
Linear independenceNot linearly independentLinearly independent set
![Page 72: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/72.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
72
Matrix rank• Column/row rank
– Column rank always equals row rank
• Matrix rank
![Page 73: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/73.jpg)
Linear Algebra Review
Stanford University
27-Sep-2
01
8
73
Matrix rank• For transformation matrices, the rank tells you the
dimensions of the output
• E.g. if rank of A is 1, then the transformation
p’=Apmaps points onto a line.
• Here’s a matrix with rank 1:
All points get mapped to the line y=2x
![Page 74: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/74.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
74
Matrix rank• If an m x m matrix is rank m, we say it’s “full rank”–Maps an m x 1 vector uniquely to another m x 1 vector– An inverse matrix can be found
• If rank < m, we say it’s “singular”– At least one dimension is getting collapsed. No way to look at
the result and tell what the input was– Inverse does not exist
• Inverse also doesn’t exist for non-square matrices
![Page 75: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/75.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
75
Outline• Vectors and matrices– Basic Matrix Operations– Determinants, norms, trace– Special Matrices
• Transformation Matrices– Homogeneous coordinates– Translation
• Matrix inverse• Matrix rank• Eigenvalues and Eigenvectors(SVD)• Matrix Calculus
![Page 76: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/76.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
76
Eigenvector and Eigenvalue• An eigenvector x of a linear transformation A is a non-zero
vector that, when A is applied to it, does not change direction.
![Page 77: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/77.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
77
Eigenvector and Eigenvalue• An eigenvector x of a linear transformation A is a non-zero
vector that, when A is applied to it, does not change direction.
• Applying A to the eigenvector only scales the eigenvector by the scalar value λ, called an eigenvalue.
![Page 78: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/78.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
78
Eigenvector and Eigenvalue• We want to find all the eigenvalues of A:
• Which can we written as:
• Therefore:
![Page 79: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/79.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
79
Eigenvector and Eigenvalue• We can solve for eigenvalues by solving:
• Since we are looking for non-zero x, we can instead solve the above equation as:
![Page 80: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/80.jpg)
Linear Algebra Review
Stanford University
27-Se
p-20
18
80
Properties• The trace of a A is equal to the sum of its eigenvalues:
• The determinant of A is equal to the product of its eigenvalues
• The rank of A is equal to the number of non-zero eigenvalues of A.
• The eigenvalues of a diagonal matrix D = diag(d1, . . . dn) are just the diagonal entries d1, . . . dn
![Page 81: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/81.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
81
Spectral theory• We call an eigenvalue λ and an associated eigenvector
an eigenpair. • The space of vectors where (A − λI) = 0 is often called
the eigenspace of A associated with the eigenvalue λ. • The set of all eigenvalues of A is called its spectrum:
![Page 82: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/82.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
82
Spectral theory• The magnitude of the largest eigenvalue (in
magnitude) is called the spectral radius
–Where C is the space of all eigenvalues of A
![Page 83: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/83.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
83
Spectral theory• The spectral radius is bounded by infinity norm of a
matrix:• Proof: Turn to a partner and prove this!
![Page 84: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/84.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
84
Spectral theory• The spectral radius is bounded by infinity norm of a
matrix:• Proof: Let λ and v be an eigenpair of A:
![Page 85: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/85.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
85
Diagonalization• An n × n matrix A is diagonalizable if it has n linearly
independent eigenvectors. • Most square matrices (in a sense that can be made
mathematically rigorous) are diagonalizable: – Normal matrices are diagonalizable –Matrices with n distinct eigenvalues are diagonalizable
Lemma: Eigenvectors associated with distinct eigenvalues are linearly independent.
![Page 86: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/86.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
86
Diagonalization
• An n × n matrix A is diagonalizable if it has n linearly independent eigenvectors.
• Most square matrices are diagonalizable: – Normal matrices are diagonalizable –Matrices with n distinct eigenvalues are diagonalizable
Lemma: Eigenvectors associated with distinct eigenvalues are linearly independent.
![Page 87: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/87.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
87
Diagonalization• Eigenvalue equation:
–Where D is a diagonal matrix of the eigenvalues
![Page 88: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/88.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
88
Diagonalization• Eigenvalue equation:
• Assuming all λi’s are unique:
• Remember that the inverse of an orthogonal matrix is just its transpose and the eigenvectors are orthogonal
![Page 89: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/89.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
89
Symmetric matrices• Properties:– For a symmetric matrix A, all the eigenvalues are real.– The eigenvectors of A are orthonormal.
![Page 90: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/90.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
90
Symmetric matrices• Therefore:
– where• So, what can you say about the vector x that satisfies the
following optimization?
![Page 91: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/91.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
91
Symmetric matrices• Therefore:
– where• So, what can you say about the vector x that satisfies the
following optimization?– Is the same as finding the eigenvector that corresponds to the
largest eigenvalue of A.
![Page 92: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/92.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
92
Some applications of Eigenvalues• PageRank • Schrodinger’s equation • PCA
• We are going to use it to compress images in future classes
![Page 93: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/93.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
93
Outline• Vectors and matrices– Basic Matrix Operations– Determinants, norms, trace– Special Matrices
• Transformation Matrices– Homogeneous coordinates– Translation
• Matrix inverse• Matrix rank• Eigenvalues and Eigenvectors(SVD)• Matrix Calculus
![Page 94: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/94.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
94
Matrix Calculus – The Gradient• Let a function take as input a matrix A of size
m × n and return a real value.• Then the gradient of f:
![Page 95: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/95.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
95
Matrix Calculus – The Gradient• Every entry in the matrix is:• the size of ∇Af(A) is always the same as the size of A. So if A
is just a vector x:
![Page 96: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/96.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
96
Exercise• Example:
• Find:
![Page 97: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/97.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
97
Exercise• Example:
• From this we can conclude that:
![Page 98: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/98.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
98
Matrix Calculus – The Gradient• Properties
![Page 99: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/99.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
99
Matrix Calculus – The Hessian• The Hessian matrix with respect to x, written or
simply as H:
• The Hessian of n-dimensional vector is the n × n matrix.
![Page 100: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/100.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
100
Matrix Calculus – The Hessian• Each entry can be written as:
• Exercise: Why is the Hessian always symmetric?
![Page 101: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/101.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
101
Matrix Calculus – The Hessian
• Each entry can be written as:
• The Hessian is always symmetric, because
• This is known as Schwarz's theorem: The order of partial derivatives don’t matter as long as the second derivative exists and is continuous.
![Page 102: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/102.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
102
Matrix Calculus – The Hessian• Note that the hessian is not the gradient of whole gradient
of a vector (this is not defined). It is actually the gradient of every entry of the gradient of the vector.
![Page 103: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/103.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
103
Matrix Calculus – The Hessian• Eg, the first column is the gradient of
![Page 104: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/104.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
104
Exercise• Example:
![Page 105: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/105.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
105
Exercise
![Page 106: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/106.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
106
Exercise
Divide the summation into 3 parts depending on whether:• i == k or• j == k
![Page 107: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/107.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
107
Exercise
![Page 108: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/108.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
108
Exercise
![Page 109: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/109.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
109
Exercise
![Page 110: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/110.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
110
Exercise
![Page 111: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/111.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
111
Exercise
![Page 112: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/112.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
112
Exercise
![Page 113: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/113.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
113
Exercise
![Page 114: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/114.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
114
Exercise
![Page 115: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/115.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
115
Exercise
![Page 116: Linear Algebra Primer - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1819/files/02_linalg_review.pdf · a ew Stanford University 27-p-2018 1 Linear Algebra Primer](https://reader035.vdocument.in/reader035/viewer/2022062414/5f087a967e708231d4223663/html5/thumbnails/116.jpg)
Linear Algebra Review
Stanford University
27-Sep-2018
116
What we have learned• Vectors and matrices– Basic Matrix Operations– Special Matrices
• Transformation Matrices– Homogeneous coordinates– Translation
• Matrix inverse• Matrix rank• Eigenvalues and Eigenvectors• Matrix Calculate