matrix lec01 intro
TRANSCRIPT
-
8/11/2019 Matrix Lec01 Intro
1/38
CSCI 5304 Fall 2014
COMPUTATIONAL ASPECTS OF MATRIX THEORY
Class time : 2:30 3:45 pm MW
Room : MechE 212
Instructor : Daniel Boley
URL : www-users.itlabs.umn.edu/classes/Fall-2014/csci5304/
September 3, 2014
Thanks to Yousef Saad for the use of his class notes.
CSci 5304: header 0-1
-
8/11/2019 Matrix Lec01 Intro
2/38
GENERAL INTRODUCTION
Linear algebra and numerical linear algebra Types of problems to be seen in this course
Mathematical background - matrices, eigenvalues, ... Quick review of Matlab
Vector norms, matrix norms
CSci 5304: BG 1-1
-
8/11/2019 Matrix Lec01 Intro
3/38
Introduction
This course is about Matrix algorithms or matrix com-
putations
It involves: algorithms for standard matrix computations(e.g. solving linear systems) - and their analysis (e.g., their
cost, numerical behavior, ..)
Matrix algorithms pervade most areas of science andengineering.
In computer science: recent increase of interest in matrix
algorithms for data mining, information retrieval, searchengines, pattern recognition, graphics, ...
CSci 5304: BG 1-2
-
8/11/2019 Matrix Lec01 Intro
4/38
Examples
Ancient Chinese Problem (3rd cent BC) [Babylonians
had similar problem]:
There are three types of corn, of which three bundles ofthe first, two of the second, and one of the third make 38
measures. Two of the first, three of the second and one of
the third make 33 measures. And one of the first, two of
the second and three of the third make 26 measures. How
many measures of corn are contained in one bundle of each
type?
CSci 5304: BG 1-3
-
8/11/2019 Matrix Lec01 Intro
5/38
Pagerank of Webpages (21st cent AD)
If one were to do a random walk from web page to webpage, following each link on a given web page at random
with equal likelihood, which are the pages to be encoun-
tered this way most often?
CSci 5304: BG 1-3a
-
8/11/2019 Matrix Lec01 Intro
6/38
Vibrations in mechanical systems. See:
www.cs.umn.edu/~saad/eig_book_2ndEd.pdf
Problem: Determine the vibration
modes of the mechanical system [to
avoid resonance]. See details inChapter 10 (sec. 10.2) of above
reference.
m1
m2
l1
l2
k1
k2
CSci 5304: BG 1-4
http://www.cs.umn.edu/~saad/eig_book_2ndEd.pdfhttp://www.cs.umn.edu/~saad/eig_book_2ndEd.pdfhttp://www.cs.umn.edu/~saad/eig_book_2ndEd.pdfhttp://www.cs.umn.edu/~saad/eig_book_2ndEd.pdf -
8/11/2019 Matrix Lec01 Intro
7/38
Examples (cont.)
Method of least-squares (inspired by first use of least
squares ever, by Gauss around 1801)
A planet follows an elliptical orbit according toay2
+bxy
+cx+ dy+e = x2 in cartesian coordinates. Given a set of
noisy observations of(x, y)positions, compute a, b, c, d, e,
and use to predict future positions of the planet. This
least squares problem is nearly rank-deficient and hence
very sensitive to perturbations in the observations.
Read Wikipedias article on planet ceres:http://en.wikipedia.org/wiki/Ceres_(dwarf_planet)
CSci 5304: BG 1-5
http://en.wikipedia.org/wiki/Ceres_(dwarf_planet)http://en.wikipedia.org/wiki/Ceres_(dwarf_planet) -
8/11/2019 Matrix Lec01 Intro
8/38
Background in linear algebra
Review vector spaces. Read section 1.1 of text on vector
notation.
A vector subspace of R
n
is a subset of R
n
that is alsoa real vector space. The set of all linear combinations of
a set of vectors G = {a1, a2, . . . , aq} of Rn is a vector
subspace called the linear span ofG,
If the ais are linearly independent, then each vector of
span{G} admits a unique expression as a linear combina-
tion of the ais. The set G is then called a basis.
Recommended reading: Sections 1.1 1.6 of
www.cs.umn.edu/~saad/eig_book_2ndEd.pdf
CSci 5304: BG 1-6
http://www.cs.umn.edu/~saad/eig_book_2ndEd.pdfhttp://www.cs.umn.edu/~saad/eig_book_2ndEd.pdfhttp://www.cs.umn.edu/~saad/eig_book_2ndEd.pdfhttp://www.cs.umn.edu/~saad/eig_book_2ndEd.pdf -
8/11/2019 Matrix Lec01 Intro
9/38
Matrices
A realmnmatrixAis anmnarray of real numbers
aij, i = 1, . . . , m, j = 1, . . . , n .
The set of all mnmatrices is a real vector space denotedby Rmn.
Complex matrices defined similarly.
A matrix represents a linear mapping between two vector
spaces of finite dimension n and m.
CSci 5304: BG 1-7
-
8/11/2019 Matrix Lec01 Intro
10/38
Operations:
Addition: C= A+ B, where A, B, C Rmn and
cij = aij+bij, i = 1, 2, . . . m, j = 1, 2, . . . n .
Multiplication by a scalar: C= A, where
cij = aij, i= 1, 2, . . . m, j = 1, 2, . . . n .
CSci 5304: BG 1-7a
-
8/11/2019 Matrix Lec01 Intro
11/38
Multiplication by another matrix:
C= AB,
where A Rmn, B Rnp, C Rmp, and
cij =
nk=1
aikbkj.
CSci 5304: BG 1-8
-
8/11/2019 Matrix Lec01 Intro
12/38
Transposition: IfA Rmn then its transpose is a matrix
C Rnm with entries
cij = aji, i= 1, . . . , n, j = 1, . . . , m
Notation : AT.
CSci 5304: BG 1-8a
-
8/11/2019 Matrix Lec01 Intro
13/38
Transpose Conjugate: for complex matrices, the trans-
pose conjugate matrix denoted by AH is more relevant:
AH = AT = AT.
CSci 5304: BG 1-9
-
8/11/2019 Matrix Lec01 Intro
14/38
Review: Matrix-matrix and Matrix-vector producs
Recall definition ofC= A B.
Recall what C represents [in terms of mappings]..
Can do the product column-wise [Matlab notation used]:
C:,j =n
k=1
bkjA:,k
Can do it row-wise:
Ci,: =n
k=1
aikBk,:
CSci 5304: BG 1-10
-
8/11/2019 Matrix Lec01 Intro
15/38
Can do it as a sum of outer-product matrices:
C=
nk=1
A:,kBk,:
Verify (prove) all 3 formulas above..
Complexity? [number of multiplications and additions]
What happen to these 3 different approaches when B
has one column (p = 1)?
CSci 5304: BG 1-11
-
8/11/2019 Matrix Lec01 Intro
16/38
Square matrices, matrix inversion, eigenvalues
Square matrix: n= m - so A Rnn
Identity matrix: square matrix with
aij = 1 ifi = j
0 otherwise
Notation: I.
Property: AI= IA = A
Inverse ofA (when it exists) is a matrix Csuch that
AC= CA= I
Notation: A1
CSci 5304: BG 1-12
-
8/11/2019 Matrix Lec01 Intro
17/38
Eigenvalues and eigenvectors
A complex scalar is an eigenvalue of the square matrixA if a nonzero vector u ofCn exists such that
Au= u.
The vector u is an eigenvector of A associated with .
The set of all the eigenvalues of A is the spectrum of A.
Notation: (A).
is an eigenvalue ofA if and only ifdet(A I) = 0
pA() = det(A I) is a polynomial of degree n in = characteristic polynomial ofA.
(A)if and only if is a root of the characteristic
polynomial pA().CSci 5304: BG 1-13
-
8/11/2019 Matrix Lec01 Intro
18/38
Spectral radius = The maximum modulus of the eigen-
values
(A) = max(A) ||.
Trace ofA = sum of diagonal elements ofA.
tr(A) = ni=1 aii. tr(A) = sum of all the eigenvalues of A counted with
their multiplicities.
CSci 5304: BG 1-13a
-
8/11/2019 Matrix Lec01 Intro
19/38
Recall that det(A) = product of all the eigenvalues of
A counted with their multiplicities.
Example: Trace, spectral radius, and determinant of
A = 2 13 0 .
CSci 5304: BG 1-14
-
8/11/2019 Matrix Lec01 Intro
20/38
For two n n matrices A and B are the eigenvalues
of AB and BA the same?
IfA is nonsingular what are the eigenvalues/eigenvectors
of A1?
What are the eigenvalues/eigenvectors ofAk
for a giveninteger power k?
What are the eigenvalues/eigenvectors of p(A) for a
polynomial p?
Review the Jordan canonical form. Define the eigen-
values, and eigenvectors from the Jordan form.
CSci 5304: BG 1-15
-
8/11/2019 Matrix Lec01 Intro
21/38
Range and null space
Range: Ran(A) ={Ax | x Rn} Rm
Null Space: Null(A) ={x Rn | Ax= 0 } Rn
Range = linear span of the columns ofA
Rank of a matrix rank(A) = dim(Ran(A)) n
CSci 5304: BG 1-15a
-
8/11/2019 Matrix Lec01 Intro
22/38
Ran(A) Rm rank (A) m
rank (A)min{m, n}
rank (A) = number of linearly independent columns of
A = number of linearly independent rows ofA
A is of full rank ifrank(A) = min{m, n}. Otherwiseit isrank-deficient.
CSci 5304: BG 1-16
-
8/11/2019 Matrix Lec01 Intro
23/38
Rank+Nullity theorem for an m n matrix:
dim(Ran(A)) +dim(Null(A)) = n
Apply toAT: dim(Ran(AT))+dim(Null(AT)) = m
rank(A) +dim(Null(AT)) = m
Terminology:
dim(Null(A)) is theNullityof A
n rank(A) is called theco-rankof A
so: co-rank ofA equals nullity ofA.
CSci 5304: BG 1-17
-
8/11/2019 Matrix Lec01 Intro
24/38
Show that A Rnn is of rank one iff [if and only if]
there exist two nonzero vectors u and v such that
A = uvT.
What are the eigenvalues and eigenvectors ofA?
Is it true that
rank(A) = rank(A) = rank(AT) = rank(AH) ?
Matlab exercise: explore the matlab function rank.
Find the range and null space of the matrix
1 1 0
1 2 3
1 2 1
2 1 1
Verify your result with matlab.CSci 5304: BG 1-18
-
8/11/2019 Matrix Lec01 Intro
25/38
CSci 5304: BG 1-19
-
8/11/2019 Matrix Lec01 Intro
26/38
Types of matrices (square)
Symmetric AT = A. Skew-symmetric AT =A.
Hermitian AH = A. Skew-Hermitian AH =A.
Normal AHA= AAH.
Nonnegative aij 0, i , j = 1, . . . , n
Similarly for nonpositive, positive, and negative matrices
Unitary QHQ= I.
CSci 5304: matrices 1-20 GvL (2.1);Heath 2.1; TB 1-2
-
8/11/2019 Matrix Lec01 Intro
27/38
[Note: Common useage restricts this definintion to com-
plex matrices an orthogonal matrix is a unitary real matrix
not very natural ]
Orthogonal QTQ= D [orthogonal columns]
The term orthonormal matrix is never used.
CSci 5304: matrices 1-21 GvL (2.1);Heath 2.1; TB 1-2
-
8/11/2019 Matrix Lec01 Intro
28/38
What is the inverse of a unitary matrix?
What can you say about the diagonal entries of a skew-
symmetric (real) matrix?
What can you say about the diagonal entries of a
Hermitian (complex) matrix? What can you say about the diagonal entries of a skew-
Hermitian (complex) matrix?
The following types of matrices are normal [true-false]:
real symmetric, real skew-symmetric, complex Hermitian,
complex skew-Hermitian.
Find all real 2 2 matrices that are normal.
Show that any triangular matrix that is normal is diag-
onalCSci 5304: matrices 1-22 GvL (2.1);Heath 2.1; TB 1-2
-
8/11/2019 Matrix Lec01 Intro
29/38
Matrices with structure
Diagonal aij = 0 for j = i. Notation :
A = diag (a11, a22, . . . , ann) .
Upper triangular aij = 0 for i > j.
Lower triangular aij = 0 for i < j.
Upper bidiagonal aij = 0 for j = i or j = i+ 1.
Lower bidiagonal aij = 0 for j = i or j = i 1.
Tridiagonal aij = 0 when |i j| >1.
CSci 5304: matrices 1-23 GvL (2.1);Heath 2.1; TB 1-2
-
8/11/2019 Matrix Lec01 Intro
30/38
Banded aij = 0 only when i ml j i + mu,
Bandwidth = ml+mu+ 1.
Upper Hessenberg aij = 0 when i > j + 1. Lower
Hessenberg matrices can be defined similarly.
Outer productA= uvT
, where bothuandvare vectors. Block tridiagonal generalizes tridiagonal matrices by
replacing each nonzero entry by a square matrix.
CSci 5304: matrices 1-24 GvL (2.1);Heath 2.1; TB 1-2
-
8/11/2019 Matrix Lec01 Intro
31/38
Special matrices
Vandermonde:
Given a column of entries[x0, x1, , xn]T put its(com-
ponent-wise)powers into the columns of a matrix V:
V =
1 x0 x
20 x
n0
1 x1 x21 x
21
... ... ... ...
1 xn x2n x
nn
Try the matlab function vander
What does the matrix-vector product V a represent?
Interpret the solution of the linear system V a = y
where a is the unknown. Sketch a fast solution method
based on this.CSci 5304: matrices 1-25 GvL (2.1);Heath 2.1; TB 1-2
-
8/11/2019 Matrix Lec01 Intro
32/38
CSci 5304: matrices 1-26 GvL (2.1);Heath 2.1; TB 1-2
T li
-
8/11/2019 Matrix Lec01 Intro
33/38
Toeplitz:
Entries are constant along diagonals, i.e., aij = rji. Determined by m+ n 1 values rji.
T =
r0 r1 r2 r3 r4
r1 r0 r1 r2 r3
r2 r1 r0 r1 r2
r3 r2 r1 r0 r1r4 r3 r2 r1 r0
T oeplitz
CSci 5304: matrices 1-26a GvL (2.1);Heath 2.1; TB 1-2
T lit t ( ) b l i O( 2)
-
8/11/2019 Matrix Lec01 Intro
34/38
Toeplitz systems (m= n) can be solver in O(n2) ops.
The whole inverse (!) can be determined inO
(n2
) ops. Explore toeplitz(c,r) in matlab.
CSci 5304: matrices 1-27 GvL (2.1);Heath 2.1; TB 1-2
-
8/11/2019 Matrix Lec01 Intro
35/38
Hankel: Entries are
constant along anti-
diagonals, i.e., aij =
hj+i1.
Determined bym+n
1 values hj+i1.
H=
h1 h2 h3 h4 h5
h2 h3 h4 h5 h6h3 h4 h5 h6 h7
h4 h5 h6 h7 h8
h5
h6
h7
h8
h9
Hankel
CSci 5304: matrices 1-27a GvL (2.1);Heath 2.1; TB 1-2
-
8/11/2019 Matrix Lec01 Intro
36/38
Circulant : Entries
in a row are cyclicallyright-shifted to form
next row. Determined
by n values.
C=
v1 v2 v3 v4 v5
v5 v1 v2 v3 v4v4 v5 v1 v2 v3
v3 v4 v5 v1 v2
v2
v3
v4
v5
v1
Circulant
CSci 5304: matrices 1-27a GvL (2.1);Heath 2.1; TB 1-2
Explore hankel(c r) in matlab
-
8/11/2019 Matrix Lec01 Intro
37/38
Explore hankel(c,r) in matlab.
How can you generate a circulant matrix in matlab?
CSci 5304: matrices 1-28 GvL (2.1);Heath 2.1; TB 1-2
S i
-
8/11/2019 Matrix Lec01 Intro
38/38
Sparse matrices
Matrices with very few nonzero entries so few thatthis can be exploited.
Many of the large matrices encountered in applications
are sparse.
Main idea of sparse matrix techniques is not to rep-
resent the zeros.
This will be covered in some detail at the end of the
course.
CSci 5304: matrices 1-29 GvL (2.1);Heath 2.1; TB 1-2