1
Systems of Linear Equations
Direct Methods
2
Solving Linear Equations
Two simultaneous equations – the solution is the intersection of two straight lines.
Three simultaneous equations – the solution is the point where three planes intersect.
3
The solution of n simultaneous linear equations with n unknowns
In matrix notation Ax = b, when
nnnnnn
n
b
b
b
x
x
x
aaa
aaa
aaa
2
1
2
1
21
222221
11111
bxA
nnnnnn
nn
nn
bxaxaxa
bxaxaxa
bxaxaxa
2211
22222121
11212111
The linear system is said to be of order n and has a unique solution if det(A) ≠ 0.
4
Two approaches for solving systems of linear equations
A. Direct Methods• Assuming negligible computational round-off errors,
direct methods solve a system of linear equations in a finite number of operations.
• These direct techniques are useful when the number of equations involved is not too large (typically of the order of 40 or fewer equations).
• Examples of direct methods: Gauss Elimination and Gauss-Jordan Elimination.
5
Two approaches for solving systems of linear equations
B. Iterative Methods• These iterative methods are more appropriate
when the number of equations involved is large (typically of the order of 100 or more), or when the matrix is sparse.
• They are more economical in memory requirements.
• Examples of iterative methods: Jacobi method and Gauss Seidel iterative methods
6
Direct Methods
1. Gauss Eliminationa. Naïve Gauss Eliminiation
b. Pivoting Strategies
c. Scaling
2. Gauss-Jordan Elimination
3. LU Decomposition
7
Gauss EliminationLinear systems of equations where the matrix is triangular are particularly easy to solve.
nnnn
nn
nn
nn
bxu
bxuxu
bxuxuxu
bxuxuxuxu
33333
22323222
11313212111
8
Exercise
33
16
86
8423
4
43
432
4321
x
xx
xxx
xxxx
What are the values of x1, x2, x3, and x4?
9
Gauss Elimination Two steps of the Gaussian Elimination
1131321211
'223
'23
'22
"33
"33
"3
'2
1
"33
'23
'22
131211
3
2
1
333231
232221
131211
/)(
/)(
/
axaxabx
axabx
abx
b
b
b
a
aa
aaa
b
b
b
aaa
aaa
aaa
Back Substitution –
To find the values of x's.
Forward Elimination –
To transform the set of equations to an upper triangular system.
10
Naive Gauss EliminationIt is called "Naïve" Gauss Elimination because it does not avoid the problem of division by zero.
"33
"33
'23
'232
'22
1313212111
3333232131
2323222121
1313212111
bxa
bxaxa
bxaxaxa
bxaxaxa
bxaxaxa
bxaxaxa
Forward EliminationPurpose: to reduce the set of equations to an upper triangular system
11
Forward Elimination
'33
'332
'32
'23
'232
'22
1313212111
3333232131
2323222121
1313212111
bxaxa
bxaxa
bxaxaxa
bxaxaxa
bxaxaxa
bxaxaxa
Step 1: Remove x1 from all equations except the first equation (which serves as the pivot equation.)
)3row (
)2row(
)1row (
3333232131
2323222121
1313212111
bxaxaxa
bxaxaxa
bxaxaxa
Let's focus on removing x1 from (row 2) first.
12
Forward Elimination
If a11 ≠ 0, define m21 = a21 / a11.
We can replace (row 2) by {(row 2) – m21 x (row 1)} to yield
1212313212321221221
121231321211121323222121
)()(0
)()(
bmbxamaxamax
bmbxaxaxamxaxaxa
1212'2132123
'23122122
'22 bmbbamaaamaa
Put
We have
)'2row (
)1row ('23
'232
'22
1313212111
bxaxa
bxaxaxa
(row 3') can be obtained in the similar fashion.
Note:a11 is called the pivot coefficient.
13
"33
"33
'23
'232
'22
1313212111
'33
'332
'32
'23
'232
'22
1313212111
bxa
bxaxa
bxaxaxa
bxaxa
bxaxa
bxaxaxa
Forward Elimination
Step 2: Remove x2 from all equations except the first two equations (row 1) and (row 2'). That is
After step 1, we have
)'3row (
)'2row (
)1row (
'33
'332
'32
'23
'232
'22
1313212111
bxaxa
bxaxa
bxaxaxa
(Basically repeat step 1 on the subsystem.)
14
Forward Elimination
If a'22 ≠ 0, define m32 = a'32 / a'22.
We can replace (row 3') by {(row 3') – m32 x (row 2')} to yield
'232
'33
'2332
'332
'232
'33
'232
'22323
'332
'32
)(0
)()(
bmbxamax
bmbxaxamxaxa
'232
'3
"3
'2332
'33
"33 bmbbamaa
Put
We have
"33
"33
'23
'232
'22
1313212111
bxa
bxaxa
bxaxaxa
15
Exercise
What are the values of a'22, a'23, a'32, a'33, b'2, b'3 ?
2
5.1
1
214
321
412
2
5.1
1
24
32
42
321
321
321
xxx
xxx
xxx
'3
'2
'33
'32
'23
'22
'3
'2
3'332
'32
3'232
'22
321 1
0
0
412142
b
b
aa
aa
b
b
xaxa
xaxa
xxx
16
Exercise (continue)
0
1
1
630
15.10
412
0
1
1
63
5.1
42
32
32
321
xx
xx
xxx
"3
"33
"33
"33
32
321
1
1
00
15.10
412
1
1
5.1
42
babxa
xx
xxx
What are the values of a"33 and b"33 ?
17
Exercise (continue)
What are the values of x1, x2, x3?
2
1
1
400
15.10
412
2
1
1
4
5.1
42
3
32
321
x
xx
xxx
18
Forward Elimination
In general, given a linear system of n equations, we repeat the procedure until the system has been transformed into an upper triangular system.
)1()1(
)1(2
)1(23
)1(232
)1(22
)0(1
)0(13
)0(132
)0(121
)0(11
332211
22323222121
11313212111
nnn
nnn
nn
nn
nnnnnnn
nn
nn
bxa
bxaxaxa
bxaxaxaxa
bxaxaxaxa
bxaxaxaxa
bxaxaxaxa
19
Back Substitution
121for )1(
1
)1()1(
)1(
)1(
,,nnia
xabx
a
bx
iii
n
ij ji
iji
i
innn
nn
n
After reducing the system of equations into an upper triangular system,
)1()1(
)2(1
)2(,11
)2(1,1
)1(2
)1(23
)1(232
)1(22
)0(1
)0(13
)0(132
)0(121
)0(11
nnn
nnn
nnn
nnnn
nnn
nn
nn
bxa
bxaxa
bxaxaxa
bxaxaxaxa
we can start solving for x's in backward manner.
20
Pseudocode to performForward Elimination
for k = 1 to n-1
for i = k+1 to n
factor = aik / akk
for j = k+1 to n
aij = aij – factor * akj
bi = bi – factor * bk
m21 = a21 / a11
1212'2 bmbb
132123'23
122122'22
amaa
amaa
21
Pseudocode to performBack Substitution
xn = bn / ann
for i = n-1 downto 1
sum = 0
for j = i+1 to n
sum = sum + aij * xj
xi = (bi – sum) / aii
121for)1(1
)1()1(
,,nnia
xabx
iii
n
ij ji
iji
i
i
)1(
)1(
n
nn
nn
n a
bx
22
Operation Count
The execution time of Gauss Elimination is measured in terms of floating-point operations (FLOPS).
23
for k = 1 to n-1
for i = k+1 to n
factor = aik / akk
for j = k+1 to n
aij = aij – factor * akj
bi = bi – factor * bk
Step k aik / akk aij – factor * akj bi – factor * bk
* and / + and - * and / + and - * and /
1 (n-1) (n-1)2 (n-1)2 (n-1) (n-1)
2 (n-2) (n-2)2 (n-2)2 (n-2) (n-2)
… … … … … …
n-2 2 22 22 2 2
n-1 1 12 12 1 1
Total n(n-1)/2 n(n-1)(2n-1)/6 n(n-1)(2n-1)/6 n(n-1)/2 n(n-1)/2
Operation Count (Forward Elimination)
6
)12)(1(
2
)1(
1
2
1
pppj
ppj
p
j
p
j
Note:
24
Operation Count (Back Substitution)
+ - * /
xn 0 0 0 1
xn-1 (i=n-1)
1 1 1 1
… … … …
x2 (i=2) (n-2) 1 n-2 1
x1 (i=1) n-1 1 n-1 1
Total n(n-1)/2 n-1 n(n-1)/2 n
xn = bn / ann
for i = n-1 downto 1
sum = 0
for j = i+1 to n
sum = sum + aij * xj
xi = (bi – sum) / aii
25
Total Number of FLOPS• Addition/Subtraction
3onsubstitutiBack
12
)1(
neliminatio Forward6
)12)(1(
2
)1( 3nn
nnnnnnn
• Multiplication/Division
33
)13(
onsubstitutiBack 2
)1(
neliminatio Forward2
)1(
2
)1(
6
)12)(1(
32 nnnn
nnnnnnnnnn
26
Total Number of FLOPS
• Total number of FLOPS – O(n3)
• For large n, the operation count for Gauss Elimination is about n3. This means that as n doubled, the cost of solving the linear equations goes up by a factor of 8.
• It is also easy to see that most of the cost of Gauss Elimination is in the elimination step.
27
Direct Methods
1. Gauss Eliminationa. Naïve Gauss Eliminiation
b. Pivoting Strategies
c. Scaling
2. Gauss-Jordan Elimination
3. LU Decomposition
28
Pitfalls of Elimination Methods
• Division by zero– When the pivot coefficient is equal to or close to zero.
Example:
• Round-off error– Round-off error propagates significantly when we
calculate the solution of a large number of equations.
• Ill-conditioned system– When the determinant is close to zero
562
3764
83
321
321
32
xxx
xxx
xx
29
(a) Singular (no solution)
(b) Singular (infinite solution)
(c) Ill-condition system (Near-singular)
Cases that cause problem
30
Ill-condition SystemWhen there is a small change in one or more of the coefficients in a system,
– Well condition system – results in a similar small change in the solution.
– Ill-condition system – results in large changes in the solution.
4.1021.1
102
21
21
xx
xx
3
4
)1.1(2)2(1)10(1.1)4.10(1
2
)1.1(2)2(1)4.10(2)10(2
1
x
x
An example of an ill-conditioned system and its solution
31
Ill-condition System
)change5%aboutonly(4.108.10)1(2)8(1.1
4.10)1(2)8(05.1
10)1(2)8(
If we change the coefficient a21 from 1.1 to 1.05, we have the equations and
the corresponding solution as
4.10205.1
102
21
21
xx
xx1
8
)05.1(2)2(1)10(1.1)4.10(1
2
)05.1(2)2(1)4.10(2)10(2
1
x
x
Substituting the solutions back to the equations,
A change of 0.05 in one of the coefficients results in a large change in the solution. However, when substituting the solutions back to the equations, we are unable to discover a large deviation.
32
Pivoting Strategies• Pivoting – making the magnitude of the diagonal
elements (pivots) as large as possible by interchanging equations.
• In Gauss Elimination, if pivot coefficient akk(k) = 0 for some
k, the method breaks down.
• Example:
122
22
1
321
321
321
xxx
xxx
xxx
This system is non-singular and has a unique solution
1321 xxx
33
However, after the 1st step in the elimination, we get
x1 + x2 + x3 = 1
x3 = 1
x2 + x3 = 0
Note that a22 = 0, we can't proceed as usual. But by
interchanging Row 2 and Row 3, we have the triangular system
x1 + x2 + x3 = 1
x2 + x3 = 0
x3 = 1
34
Pivoting Strategies
• Partial Pivoting– Interchange equations (rows) only– Choose r, the smallest integer for which
i.e. choose the largest coefficient in the column to become the pivot coefficient.
– interchange rows k and r
nikaa kik
krk )()( max
35
Pivoting Strategies
• Complete Pivoting– Interchange both equations and unknown
elements (pivots)– Choose r and s, the smallest integer for which
i.e. choose the largest coefficient in the sub-matrix to become the pivot coefficient.
– interchange rows k and r, and columns k and s.
njikaa kij
krs ,max )()(
36
Exercise (Partial Pivoting)
Suppose k = 2 (2nd iteration). Which row should be selected as the pivot row?
5
4
3
2
1
100580
2992560
16540
1523300
71521
2
4
3
5
1
1523300
2992560
16540
100580
71521
37
Exercise (Complete Pivoting)
Suppose k = 2 (2nd iteration). Which row should be selected as the pivot row?
5
4
3
2
1
100580
2882560
16540
1523300
719921
5
2
3
4
1
108500
1503320
14560
2625880
729911
38
Example (Without partial pivoting)• Use Gauss elimination to solve
0000.10000.10000.1
0001.20000.30003.0
21
21
xx
xx(Ans: x1 = 1/3, x2 = 2/3)
• Note that the first pivoting element a11 is close to 0.• Forward elimination gives
• Backward substitution gives
66669999
0001.20000.30003.0
2
21
x
xx
0003.0
)3/2(30001.2,
3
212
xx
39
Significant
Figuresx2 x1
x1
|εt| (%)3 0.667 -3.33 1099
4 0.6667 0.0000 100
5 0.66667 0.30000 10
6 0.666667 0.330000 1
7 0.6666667 0.3330000 0.1
The solution is sensitive to the number of significant figures in the computation, since we are subtracting two almost equal numbers.
(Ans: x1 = 1/3, x2 = 2/3)
Error!
40
Example (With partial pivoting)• Reverse order
0001.20000.30003.0
0000.10000.10000.1
21
21
xx
xx(Ans: x1 = 1/3, x2 = 2/3)
• Forward elimination gives
• Backward substitution gives
66669999
0000.10000.10000.1
2
21
x
xx
3
1
1
)3/2(1,
3
212
xx
41
Significant
Figuresx2 x1
x1
|εt| (%)3 0.667 0.333 0.1
4 0.6667 0.3333 0.01
5 0.66667 0.33333 0.001
6 0.666667 0.333333 0.0001
7 0.6666667 0.3333333 0.00001
The solution is less sensitive to the number of significant figures in the computation.
(Ans: x1 = 1/3, x2 = 2/3)
Error!
42
Singularity• Now consider x1 + x2 + x3 = 4
2x1 – x2 – x3 = 7
x1 – 2x2 – 2x3 = 12
x1 + x2 + x3 = 4
– 3x2 – 3x3 = -1
– 3x2 – 3x3 = 8
x1 + x2 + x3 = 4
– 3x2 – 3x3 = -1
0x3 = 9We can detect singularity during the elimination process.
43
Summary of Pivoting
• How to perform partial and complete pivoting
• Reasons for performing pivoting– Avoid division by zero– Minimize the effect of rounding error
44
Direct Methods
1. Gauss Eliminationa. Naïve Gauss Eliminiation
b. Pivoting Strategies
c. Scaling
2. Gauss-Jordan Elimination
3. LU Decomposition
45
Scaling
Rescale all coefficients in a row to make the largest coefficient equal to 1.
Example:
100002.0000,100000,1002 2121 xxxx
46
Questions
• What is the effect of scaling?
• What do we need to scale?
47
Example of Scaling• Use Gauss Elimination to solve
000,100000,1002
2
21
21
xx
xx
(Ans: x1 = 1.00002, x2 = 0.99998)
48
Example of Scaling (Assume 3 significant digits)
• Solution (a): Without pivoting and scaling
000,100000,1002
2
21
21
xx
xx
x2 = 1.00, x1 = 1.00 (Right)
000,100000,100
2
2
21
x
xx
• Forward elimination gives
• Back substitution gives
49
• Solution (b): With partial pivoting but no scaling
2
000,100000,1002
21
21
xx
xx
x2 = 1.00, x1 = 0.00 (Wrong)
• Forward elimination gives
• Back substitution gives
000,50000,50
000,100000,1002
2
21
x
xx
Example of Scaling (Assume 3 significant digits)
50
• Solution (c): With partial pivoting and scaling
100002.0
2
21
21
xx
xx
x2 = 1.00, x1 = 1.00 (Right)
• Forward elimination gives
• Back substitution gives
00.1
2
2
21
x
xx
Example of Scaling (Assume 3 significant digits)
51
Observation• Scaling was useful in determining whether
pivoting was necessary.
• Scaling introduces round-off error. Example,
• It is sometimes suggested that the scaling values should be used to determining the pivoting, but the original coefficient values are retained for the actual elimination and substitution computation.
00000333.000000667.0Scaling
1000,3002
21
21
xx
xx
52
Pseudocode for Gauss Elimination with Partial Pivoting// Assume arrays start with index 1 instead of 0.
// a: Coef. of matrix A; 2-D array// b: Coef. of vector b; 1-D array// n: Dimension of the system of equations// x: Coef. of vector x (to store the solution)// tol: Tolerance; smallest possible scaled pivot allowed.// er: Pass back -1 if matrix is singular. (Reference var.) Gauss(a, b, n, x, tol, er) { Declare s[n] // as an n-element array for storing scaling factors
// si = the largest coef. of row i. for i = 1 to n
si = abs(ai,1) for j = 2 to n
if (abs(ai,j) > si)
si = abs(ai,j)
Eliminate(a, s, n, b, tol, er) // Forward Elimination if (er ≠ -1) // If not singular Substitute(a, n, b, x) // Back Substitution}
53
Pseudocode for Gauss Elimination with Partial Pivoting
// Partial Pivoting
Pivot(a, b, s, n, k) {
p = k // Assume row k is the pivot row
// Find the largest scaled coefficient in column k
big = abs(ak,k / sk)
for i = k+1 to n {
dummy = abs(ai,k / si)
if (dummy > big) {
big = dummy
p = i // Record new pivoting row
}
}
// Next: Swap row p and row k if p != k
// Continue next page
54
Pseudocode for Gauss Elimination with Partial Pivoting if (p != k) {
// Swap row p and row k for j = k to n {
dummy = ap,j
ap,j = ak,j
ak,j = dummy }
// swap bp and bk
dummy = bp
bp = bk
bk = dummy
// swap sp and sk
dummy = sp
sp = bk
sk = dummy }}
55
Pseudocode for Gauss Elimination with Partial PivotingEliminate(a, b, n, x, tol, er) {
for k = 1 to n-1 { Pivot(a, b, s, n, k) // Partial Pivoting
if (abs(ak,k/sk) < tol) { // Check for singularity er = -1; return; }
for i = k+1 to n
factor = ai,k / ak,k
for j = k+1 to n
ai,j = ai,j – factor * ak,j
bi = bi – factor * bk
}
if (abs(an,n/sn) < tol) // Check for singularity er = -1;}
Forward Elimination
56
Pseudocode for Gauss Elimination with Partial Pivoting
// Back Subsitution
Substitute(a, n, b, x) {
xn = bn / an,n
for i = n-1 downto 1 {
sum = 0
for j = i+1 to n
sum = sum + ai,j * xj;
xi = (bi – sum) / ai,i
}
}
57
Question
• Why don't we replace the coefficients by the scaled values in the pseudo code?
• Note: In the pseudo code, the equation are not scaled, but scaled values of the elements are used to determine whether pivoting is to be implemented.
58
Summary of Scaling
• Scaling gives us a "true picture" which element is more dominating so that we can correctly carry out pivoting.
• Why do we need to perform pivoting again?
59
Summary
• Gauss Elimination– Forward Elimination and its complexity– Back substitution and its complexity
• Effect of pivoting and scaling