solving linear systems:direct methods - unc charlotte · matrix condition number defined for...
TRANSCRIPT
Solving Linear Systems:Direct Methods
� A number of engineering problems or models can be formulated interms of systems of equations
� Examples: Electrical Circuit Analysis, Radiative Heat Transfer (orequivalently, global diffuse illumination compuation), etc.
� Need a variety of computational techniques, depending on natureof system (sparse systems, for instance, benefit from iterative solu-tions).
� Systems can be very large, thousands of equations is not uncom-mon.
� Large systems require computational/numerical solutions.
ITCS 4133/5133: Intro. to Numerical Methods 1 Solving Linear Systems
Example: Current in an Electrical Circuit
ITCS 4133/5133: Intro. to Numerical Methods 2 Solving Linear Systems
Solving Linear Systems: Direct Methods
� Basic Gaussian Elimination (GE)
� Gaussian Elimination with Row Pivoting
� Tridiagonal Systems (Gauss-Thomas Method)
� Gauss-Jordan Elimination
ITCS 4133/5133: Intro. to Numerical Methods 3 Solving Linear Systems
Linear Systems of Equations
a11X1 +a12X2 + · · · +a1nXn = C1a21X1 +a22X2 + · · · +a2nXn = C2
... ... ... ... ...an1X1 +an2X2 + · · · +annXn = Cn
where aijs are the coefficients, Xjs are the unknowns, and Cis are theknown constants.
Assumptions
⇒ Number of unknowns equal the number of equations
⇒ Equations are linearly independent
ITCS 4133/5133: Intro. to Numerical Methods 4 Solving Linear Systems
Solution of Small Systems of Equations
For small systems such as
a11X1 + a12X2 = C1
a21X1 + a22X2 = C2
can be solved by substitution
⇒ Complex engineering problems could contain thousands of equa-tions.
⇒ Substitution is not a feasible technique for large systems
ITCS 4133/5133: Intro. to Numerical Methods 5 Solving Linear Systems
Linear System Classification
� With solutions
� Without solutions
� With infinite solutions
ITCS 4133/5133: Intro. to Numerical Methods 6 Solving Linear Systems
Gaussian Elimination
� Forward Pass: convert the system of equations into an upper trian-gular matrix,
� Back Substitution: to solve the system.
Permissible Operations
⇒ Changing the order of the equations does not change the solution
⇒ An equation may be scaled (multiplied/divided) by a non-zero con-stant without changing the solution
⇒ May add 2 equations together and replace either one with the newequation, termed (linear combination)
ITCS 4133/5133: Intro. to Numerical Methods 7 Solving Linear Systems
Gaussian Elimination:Matrix Representation
a11 a12 · · · a1n
a21 a22 · · · a2n... ... ... ...an1 an2 · · · ann
X1X2...Xn
=
V1V2...Vn
or,
a11 a12 · · · a1n C1a21 a22 · · · a2n C2... ... ... ... ...an1 an2 · · · ann Cn
ITCS 4133/5133: Intro. to Numerical Methods 8 Solving Linear Systems
Gaussian Elimination:Forward Pass
Results in the upper triangular matrix:1 d12 d13 · · · d1n e10 1 d23 · · · d2n e20 0 1 · · · d3n e3... ... ... ... ... ...0 0 0 · · · 1 en
ITCS 4133/5133: Intro. to Numerical Methods 9 Solving Linear Systems
Gaussian Elimination:Back Substitution
X1 + d12X2 + d13X3 + · · · + d1,n−1Xn−1 + d1nXn = e1
X2 + d23X3 + · · · + d2,n−1Xn−1 + d2nXn = e2
X3 + · · · + d3,n−1Xn−1 + d3nXn = e3
Xn−2 + dn−2,n−1Xn−1 + dn−2,nXn = en−2
Xn−1 + dn−1,nXn = en−1
Xn = en
ImplementationDiagonalize the matrix, resulting in the solution vector in the last column.
ITCS 4133/5133: Intro. to Numerical Methods 10 Solving Linear Systems
Gaussian Elimination:Algorithm
Forward Pass:
⇒ Loop over each row i, making it the pivot row
⇒ Normalize the pivot row, i (pivot element becomes 1)
aij = aij/aii, j = i + 1, i + 2, ..., (n + 1)
Ci = Ci/aii
aii = 1
⇒ Loop over rows (i + 1) to n and reduce elements in each row.
akj = akj − akiaij, j = i, ..., n
Ck = Ck − akiCi, k = (i + 1), ..., n
ITCS 4133/5133: Intro. to Numerical Methods 11 Solving Linear Systems
Gaussian Elimination:Algorithm
Forward Pass (Alternate)
� Uses a multiplier, without normalizing pivot element
Input: aij, bi, nColumn: for k = 1 to n− 1
for i = k + 1 to nMultiplier: mik = aik/akk
Row: for j = k + 1 to naij = aij −mikakj
endbi = bi −mikbk
endend
Output: Upper Triangular Matrix
ITCS 4133/5133: Intro. to Numerical Methods 12 Solving Linear Systems
Gaussian Elimination:Algorithm
Back Substitution:
⇒ For the last row, Xn = Cn
⇒ For rows (n− 1) through 1,
Xi = Ci −n∑
j=i+1
aijXj, j = (i + 1), ..., n and i = (n− 1), ..., 1
ITCS 4133/5133: Intro. to Numerical Methods 13 Solving Linear Systems
Gaussian Elimination: Algorithm
ITCS 4133/5133: Intro. to Numerical Methods 14 Solving Linear Systems
Example 1:Electrical Circuit
ITCS 4133/5133: Intro. to Numerical Methods 15 Solving Linear Systems
Roundoff Errors and Pivoting
� Truncation and Roundoff errors can occur when coefficients differ byorders of magnitude.
� To minimize errors, diagonal elements should contain the largestcoefficient
� This can be accomplished by swapping rows during Gaussian elim-ination, a process called partial pivoting
� Zero Pivot Element: rearrange equations to prevent zero coefficientsin the pivot element
� Equations must be linearly independent, else we have a singularcoefficient matrix
ITCS 4133/5133: Intro. to Numerical Methods 16 Solving Linear Systems
GE with Pivoting: Motivation
◦ Consider the following example:
ITCS 4133/5133: Intro. to Numerical Methods 17 Solving Linear Systems
GE with Pivoting: Algorithm
ITCS 4133/5133: Intro. to Numerical Methods 18 Solving Linear Systems
GE with Pivoting: Example
ITCS 4133/5133: Intro. to Numerical Methods 19 Solving Linear Systems
Banded Systems
� Systems of equations with banded structure, with non-zero elementson the diagonal and a small number of off-diagonals.
� Such systems can be solved more efficiently.
� Example: Tridiagonal systemd1x1 a1x2 = r1b2x1 d2x2 a2x3 = r2. . .. . .
bn−1xn−2 dn−1xn−1 an−1xn = rn−1bnxn−1 dnxn = rn
� Both memory and computational savings!
ITCS 4133/5133: Intro. to Numerical Methods 20 Solving Linear Systems
Gauss-Thomas Method
� Representation:Vectors d (diagonal), a (above diagonal elements),b (below diagonal elements)
� Example:
ITCS 4133/5133: Intro. to Numerical Methods 21 Solving Linear Systems
Gauss-Thomas Method (contd)
ITCS 4133/5133: Intro. to Numerical Methods 22 Solving Linear Systems
Gauss-Thomas Method:Algorithm
ITCS 4133/5133: Intro. to Numerical Methods 23 Solving Linear Systems
Gauss-Jordan Elimination
� In Gaussian elimination, we create upper triangular matrix, followedby back-substitution
� In Gauss-Jordan elimination, the input matrix is diagonalized in theforward pass, completing the solution
� More efficient when the same system is used to solve multiple right-side vectors.
ITCS 4133/5133: Intro. to Numerical Methods 24 Solving Linear Systems
Matrix Condition Number
� Defined for non-singular matrices
� Defined in terms of the norm of the matrix, ||A||� Can be derived from vector norms:
||x||1 = |x1| + |x2| + ... + |xn|||x||2 = (|x1|2 + |x2|2 + ... + |xn|2)
1/2
||x||∞ = Max|xi|
� Matrix norms are defined as
||A||1 = max(over j) [sum of column j]||A||∞ = max (over i) [sum of row i]||A||2 = ( Max Eigen value of AHA)1/2
ITCS 4133/5133: Intro. to Numerical Methods 25 Solving Linear Systems
Matrix Condition Number (contd)
� Ill-conditioned: Sensitive to small changes in parameters definingthe problem.
� In general, condition number is defined as
κ(A) = ||A||||A−1||
� Spectral condition number:
κ(A) = ||A2||||A−12 ||
� Relative change in solution : depends on product of condition num-ber and relative change in RHS vector (b).
ITCS 4133/5133: Intro. to Numerical Methods 26 Solving Linear Systems
Analysis: Gaussian Elimination
Why does Gaussian Elimination work?
� Equation forming a linear combination of two equations also passesthrough a point common (intersection) to both equations.
◦ Consider
S1 : a0 + a1x1 + . . . + anxn = 0
T1 : a0 + b1x1 + . . . + bnxn = 0
◦ If r = (r1, r2, . . . , rn) is a solution vector it also satisfies m1S1 +m2T1
◦ Elimination produces a system solvable by back substitution.
ITCS 4133/5133: Intro. to Numerical Methods 27 Solving Linear Systems
Analysis: Gaussian Elimination -Computational Complexity
� Measure Flops.
� Row calculations: multiplier calculation (1) + n multiplications and n ad-ditions + RHS update
� Repeat for (n − 1) rows, resulting in (n+1)(n-1) multiplications and divi-sions, n(n-1) additions
k=n−1∑k=1
(n− k + 1)(n− k) =
k=n−1∑k=1
n2 − 2nk + k2 + n− k
=
k=n−1∑k=1
(n2 + n)−k=n−1∑
k=1
(2n + 1)k +
k=n−1∑k=1
k2
=n3
3− n
3
ITCS 4133/5133: Intro. to Numerical Methods 28 Solving Linear Systems
Analysis: GE with Pivoting
� Some form of row scaling can also be used.
� However, ill-conditioned matrices are not helped by scaling.
� Scaling is difficult to incorporate in general purpose packages.
� Computational complexity is O(n3), so preferable to use more efficientmethods.
ITCS 4133/5133: Intro. to Numerical Methods 29 Solving Linear Systems
Analysis: GT Method (Tridiagonal Systems)
� First equation: 2 divisions (scaling the equation)
� Next (n-2) equations: 2 mults, 2 divisions.
� Last equation: 2 mults, 1 division
� Total (Forward Step): 5 + 4(n− 2)
� Back Substitution: (n-1) multiplies.
ITCS 4133/5133: Intro. to Numerical Methods 30 Solving Linear Systems