linear algebra and complexity chris dickson cas 746 - advanced topics in combinatorial optimization...
TRANSCRIPT
Linear Algebra and Complexity
Chris DicksonCAS 746 - Advanced Topics in Combinatorial
Optimization
McMaster University, January 23, 2006
2
Review of linear algebra Complexity of determinant, inverse, systems of equations, etc Gaussian elimination method for soliving linear systems
complexity solvability numerical issues
Iterative methods for solving linear systems Jacobi Method Gauss-Sidel Method Successive Over-Relaxation Methods
Introduction
3
Review
A subset L of Rn (respectively Qn) is called a linear hyperplane if L = { x | ax = 0 } for some nonzero row vector a in Rn (Qn).
A set U of vectors in Rn (respectively Qn) is called a linear subspace of Rn if the following properties hold: The zero vector is in U If x U then x U for any scalar If x,y U then x + y U
Important subspaces Null Space : null(A) = { x | Ax = 0 } Image : im(A) = { b | Ax = b for some x in A }
A basis for a subspace L is a set of linearly independent vectors that spans L
4
Review
Theorem 3.1. Each linear subspace L of Rn is generated by finitely many vectors and is also the intersection of finitely many linear hyperplanes
Proof : If L is a subspace, then a maximal set of linearly independent vectors
from L will form a basis. Also, L is the intersection of hyperplanes { x | aix = 0 } where the set of
all ai are linearly independent generating L* := { z | zx = 0 for all x in L }
5
Review
Corollary 3.1a. For each matrix A there exist vectors x1 , … , xT such that: Ax = 0 x = 1x1 + … + TxT for certain rationals 1 , … T.
If x1 , … , xT are linearlly independent they form a fundamental system of solutions to Ax = 0.
Corollary 3.1b The system Ax = b has a solution iff yb = 0 for each vector y with yA = 0. [ Fundamental theorem of linear algebra, Gauss 1809].
6
Review
Proof : () If Ax=b then when yA = 0 we have yAx = 0 yb = 0. ()Let im(A) = {z | Ax = z for some x }. By Thrm 3.1 there
exists a matrix C such that im(A) = { z | Cz = 0 }. So, CAx = 0 implies that CA is all-zero. Thus:
(yA = 0 yb = 0) Cb = 0 b im(A) There is a solution to Ax = b
7
Review
Corollary 3.1c. If x0 is a solution to Ax = b, then there are vectors x1, … , xt, such that Ax = b iff x = x0 + 1x1 + … + txt for certain rationals 1, … , t.
Proof: Ax = b iff A(x – x0) = 0. Cor 3.1a implies:
(x – x0) = 1x1 + … + txt
x = x0 + 1x1 + … + txt
8
Review
Cramer’s Rule: If A is an invertible (n x n)-matrix, the solution to Ax = b is given by:
x1 = detA1 / detA…
xn = detAn / detA
Ai is defined by replacing the ith column of A by the column vector b.
9
Review
Example:
5x1 +x2 – x3 = 49x1 +x2 – x3 = 1 x1 -x2 +5x3 = 2
Here, x2 = det(A2) / det(A) = 166 / 16 = 10.3750
10
Sizes and Characterizations
Rational Number: r = p / q (p Z, q N, relatively prime)
Rational Vector c = ( 1 , 2 , ... , n )
Rational Matrix
We only consider rational entries for vectors and matrices.
11
Sizes and Characterizations
Theorem 3.2. Let A be a square rational matrix of size . Then, the size of det(A) is less than 2 .
Proof: Let A = (pij/qij) for i,j = 1..n. Let detA = p/q where p,q and pij ,qij are all relatively prime. Then:
From the definitition of the determinant:
12
Sizes and Characterizations
Noting that |p| = |detA|q, we combine the two formulas to bound p and q:
Plugging the bounds for p and q into the definition for the size of a rational number, we get:
13
Sizes and Characterizations
Several important corollaries from the theorem
Corollary 3.2a. The inverse A-1 of a nonsingular (rational) matrix A
has size polynomially bounded by the size of A. Proof: Entries of A-1
are quotients of subdeterminants of A. Then, apply the theorem.
Corollary 3.2b. If the system Ax=b has a solution, it has one of size polynomially bounded by the sizes of A and bProof: Since A-1 is polynomially bounded, then A-1b is polynomially bounded.
This tells us that the Corollary 3.1b provides a good characterization
14
Sizes and Characterizations
What if there isn’t a solution to Ax = b? Is this problem still polynomially bounded? Yes!
Corollary 3.2c says so! If there is no solution, we can specify a vector y with yA = 0 and yb = 1. By corollary 3.2b, such a vector exists and is polynomially bounded by the sizes of A and b.
Gaussian elimination provodes a polynomial method for finding (or not finding) a solution
One more thing left to show. Do we know the solution will be polynomially bounded in size?
15
Sizes and Characterizations
Corollary 3.2d. Let A be a rational m x n matrix and let b be a rational column vector such that each row of the matrix [A b] has size at most . If Ax = b has a solution, then:
{ x | Ax = b} = { x0 + 1x1 + txt | i R } (*)
for certain rational vectors x0 , .... , xt of size at most 4n2 . Proof: By Cramer’s Rule, there exists x0 , ... , xt satisfying (*) which
are quotients of subdeterminants of [A b] of order at most n. By theorem 3.2, each determinant has size <= 2n. Each component of xi has size less than 4n. Since each xi has n components, the size of xi is at most 4n2.
16
Gaussian Elimination
Given a system of linear equations:
a11x1 + a12x2 + .... + a1nxn = b1
a21x1 + a22x2 + .... + a2nxn = b2
.........an1x1 + an2x2 + .... + annxn = bn
We subtract appropriate multiples of the first equation from the other equations to get:a11x1 + a12x2 + .... + a1nxn = b1
a’22x2 + .... + a’2nxn = b2
......... a’n2x2 + .... + a’nnxn = bn
We then recursively solve the system of the last m-1 equations.
17
Gaussian Elimination
Operations allowed on matrices: Adding a multiple of one row to another row Permuting rows or columns
Forward Elimination: Given a matrix Ak of the form
We choose a nonzero component of D called the pivot element. Without loss of generality, assume that this pivot element is in D11. Next, add rational multiples of the first row of D to the other rows in D such that D11 is the only non-zero element in the first column of D.
(B is non-singular, upper triangular, order k)
18
Gaussian Elimination
Back Substitution: After FE stage, we wish to transform the matrix into the form:
where is non-singular diagonal.
We accomplish this by adding multiples of the kth row to the rows
1, ... , k-1 such that the element Akk is the only non-zero in the kth row and column for k = r, r-1, ... , 1. (r = rank(A))
19
Gaussian Elimination
Theorem 3.3. For rational data, the Gaussian elminination method is a polynomial time method. proof is found in the book. It is very longwinded! since operations allowed for GE are polynomially bounded, it only
remains to show that numbers do not grow exponentially. Corollary of the theorem implies that other operations on matrices
are polynomially solvable as they depend on solving a system of linear equations. calculating determinant of a rational matrix determining the rank of a rational matrix calculating the inverse of a rational matrix testing a set of vectors for linear independence solving a system of rational linear equations
20
Gaussian Elimination
Numerical Considerations .A naive pivot strategy always uses D11 as the pivot. However, this can
lead to numerical problems when we do not have exact arithmetic. (ie/ floating point numbers).
Partial pivoting always selects the largest element (in absolute value) in the first column of D as the pivot. (swap rows to make this D11)
Scaled partial pivoting is like partial pivoting, but it scales the pivot column and row so that D11 = 1.
Full pivoting uses the largest value in the entire D submatrix as the pivot. We then swap rows and columns to put this value in D11.
Studies show that full pivoting usually requires more overhead to be beneficial.
21
Gaussian Elimination
Other methods: Gauss-Jordan Method: This method combines the FE and BS stages in
Gaussian elimination. inferior to ordinary Gaussian elimination
LU Decomposition: Uses GE to decompose original matrix A into the product of a lower and upper triangular matrix.
This is the method of choice when we must solve many systems with the same matrix A but with different b’s.
We only need to perform FE once, then do back substitution for each right hand side.
Iterative Methods: Want to solve Ax = b by determining a sequence of vectors x0 , x1, ... , which eventually converge to a solution x*.
22
Iterative Methods
Given Ax = b and x0, the goal is find a sequence of iterates x1, x2 , ... , x* such that Ax* = b.
For most methods, the goal is to split A into A = L + D + U where L is strictly lower triangular, D is diagonal, and U is strictly upper triangular.
Example: L D U
23
Iterative Methods
Now, we can define how the iteration proceeds. In general:
Jacobi Iteration: Mj = D , Nj = -(L + U). Alternatively, the book presents the iteration:
These two are equivalent as book assumes aii = 1, which means D = D-1 = I. (Identity)
24
Iterative Methods
Example: Solve for x in Ax=b using Jacobi’s Method where:
Using the iteration in the previous slide and x0 = b, we obtain:
x1 = [ -5 0.667 1.5 ]T
x2 = [ 0.833 -1.0 -1.0]T .........x10 = [0.9999 0.9985 1.9977 ]T [ x* = [ 1 1 2] T ]
25
Iterative Methods
Example 2: Solve for x in Ax=b using Jacobi’s Method where:
Using x0 = b, we obtain:
x1 = [ -44.5 -49.5 -13.4 ]T
x2 = [ 112.7 235.7 67.7]T .........x10 = [2.82 4.11 1.39 ]T *106 [ x* = [ 1 1 1] T ]
26
Iterative Methods
• .What happened? The sequence diverged!
• The matrix in the second example was NOT diagonally dominant.
• Strict diagonal row dominance is sufficient for convergence. • Diagonal element is greater than sum of all other elements in the same
row (take absolute values!)• Not neccessary though as example 1 did not have strict diagonal
dominance.
• Alternatively, book says that Jacobi Method will converge if all eigenvalues of (I - A) are less than 1 in absolute value. • Again, this is a sufficient condition. It can be checked that eigenvalues
of matrix in first example do not satisfy this condition.
27
Iterative Methods
Gauss-Sidel Method: Computes the (i+1)th component of xk+1. ie,
where
28
Iterative Methods
Gauss-Sidel Method: Alternatively, if we split A = L+D+U, then we obtain the iteration:
Where MG = D + L , and NG = -U.
Again, it can be shown that the book’s method is equivalent to this method.
29
Iterative Methods
Example: Gauss-Sidel Method
Starting with x0 = b, we obtain the sequence:
x1 = [ -4.666 -5.116 3.366 ]T
x2 = [ 1.477 -2.788 1.662]T x3 = [ 1.821 -0.404 1.116]T
....
x12 = [1.000 1.000 1.000 ]T
30
Iterative Methods
It should also be noted that in both GS and Jacobi Methods, the choice of M is important.
Since the iterations of each method involve inverting M, we should choose M so that this is easy.
In Jacobi Method, M is a diagonal matrix Easy to invert!
In Gauss-Sidel Method, M is a lower triangular Slightly harder. Although inverse is still lower triangular, we must perform a little
more work, but we only have to invert it once. Alternatively, use the strategy by the book to compute iterates component by
component.
31
Iterative Methods
Successive Over-Relaxation Method (SOR) Write matrices L and U (L lower triangular with diagonal) such that:
for some appropriately chosen > 0 ( is the diagonal of A)
A related class of iterative relaxation methods for approximating the solution to Ax = b can now be defined.
32
Iterative Methods
Algorithm:
For a certain with 0<≤2 convergent if 0<<2 and Ax = b has a solution.
33
Iterative Methods
There are many different choices that can be made for the relaxation methods
Choice of = 2, then xk+1 is reflection of xk into xi = = 1, then xk+1 is projection of xk onto the hyperplane xi =
How to choose violated equation Is it best to just pick any violated equation? Should we have a strategy? (First, Best, Worst, etc)
Stopping Criteria: How close is close enough?
34
Iterative Methods
Why use iterative methods? Sometimes they will be more numerically stable. Particulary when matrix A is positive-semi-definite and diagonally
dominant. (This is the case for the linear least squares problem) If an exact solution is not neccessary. Only need to be ‘close enough’.
Which one to use? Compared with Jacobi Iteration, Gauss-Seidel Method converges faster,
usually by a factor of 2.
35
References
Schrijver, A. Theory of Linear and Integer Programming. John Wiley & Sons, 1998.
Qiao, S. Linear Systems. CAS 708 Lecture Slides. (2005)
36
THE END