cise301_topic3kfupm1 se301: numerical methods topic 3: solution of systems of linear equations...
TRANSCRIPT
CISE301_Topic3 KFUPM 1
SE301: Numerical Methods
Topic 3: Solution of Systems of Linear
Equations Lectures 12-17:
KFUPM
Read Chapter 9 of the textbook
CISE301_Topic3 KFUPM 2
Lecture 12Vector, Matrices, and
Linear Equations
CISE301_Topic3 KFUPM 3
VECTORS
1
0
0
0
,
0
1
0
0
,
0
0
1
0
,
0
0
0
1
1
2tor column vec 241 vector row
:Examples
numbers ofarray ldimensiona one a:Vector
4321 eeeevectorsIdentity
CISE301_Topic3 KFUPM 4
MATRICES
1200
1410
0143
0021
lTridiagona,
6000
0000
0040
0001
diagonal
10
01matrix identity
0
0
00
00matrix zero
:Examples
numbers ofarray ldimensiona twoa:Matrix
CISE301_Topic3 KFUPM 5
MATRICES
1000
1400
0140
3121
ngular upper tria,
451
501
112
symmetric
:Examples
CISE301_Topic3 KFUPM 6
Determinant of a MATRICES
82)015(1)512(1)25(2
50
1-31-
45
1-31-
45
502
451
501
132
det
:Examples
only matrices squarefor Defined
CISE301_Topic3 KFUPM 7
Adding and Multiplying Matrices
ji
ji
m
k
,bacBAC *
pm ifonly defined is ABCproduct The *
q)B(p and m)(nA matrices twooftion Multiplica
,bacBAC *
size same thehave they ifonly Defined *
B andA matrices twoofaddition The
1kjikij
ijijij
CISE301_Topic3 KFUPM 8
Systems of Linear Equations
formMatrix form Standard
7
5
3
601
315.2
342
76
535.2
3342
formsdifferent in
presented becan equationslinear of systemA
3
2
1
31
321
321
x
x
x
xx
xxx
xxx
CISE301_Topic3 KFUPM 9
Solutions of Linear Equations
52
3
:equations following theosolution t a is 2
1
21
21
2
1
xx
xx
x
x
CISE301_Topic3 KFUPM 10
Solutions of Linear Equations A set of equations is inconsistent if there
exists no solution to the system of equations:
ntinconsiste are equations These
542
32
21
21
xx
xx
CISE301_Topic3 KFUPM 11
Solutions of Linear Equations Some systems of equations may have infinite
number of solutions
allfor solution ais)3(5.0
solutions ofnumber infinite have
642
32
2
1
21
21
aa
a
x
x
xx
xx
CISE301_Topic3 KFUPM 12
Graphical Solution of Systems ofLinear Equations
52
3
21
21
xx
xx
solution
CISE301_Topic3 KFUPM 13
Cramer’s Rule is Not Practical
way efficient in computed are tsdeterminan theif used becan It
system. 30by 30 a solve toyears10 needscomputer super A
. systems largefor practicalnot is Rule sCramer'
2
21
11
51
31
,1
21
11
25
13
system thesolve toused becan Rule sCramer'
17
21 xx
CISE301_Topic3 KFUPM 14
Naive Gaussian Elimination Examples
Lecture 13 Naive Gaussian
Elimination
CISE301_Topic3 KFUPM 15
Naive Gaussian Elimination The method consists of two steps:
Forward Elimination: the system is reduced to upper triangular form. A sequence of elementary operations is used.
Backward Substitution: Solve the system starting from the last variable.
'
'
'00
''0
3
2
1
3
2
1
33
2322
131211
3
2
1
3
2
1
333231
232221
131211
b
b
b
x
x
x
a
aa
aaa
b
b
b
x
x
x
aaa
aaa
aaa
CISE301_Topic3 KFUPM 16
Elementary Row Operations
Adding a multiple of one row to another
Multiply any row by a non-zero constant
CISE301_Topic3 KFUPM 17
ExampleForward Elimination
18
27
6
16
14320
18120
2240
4226
4 3, 2,equationsfrom Eliminate:Step1
nEliminatio Forward:1Part
34
19
26
16
18146
39133
106812
4226
4
3
2
1
1
4
3
2
1
x
x
x
x
x
x
x
x
x
CISE301_Topic3 KFUPM 18
ExampleForward Elimination
3
9
6
16
3000
5200
2240
4226
4equation from Eliminate:Step3
21
9
6
16
13400
5200
2240
4226
4 ,3equationsfrom Eliminate:Step2
4
3
2
1
3
4
3
2
1
2
x
x
x
x
x
x
x
x
x
x
CISE301_Topic3 KFUPM 19
ExampleForward Elimination
3
9
6
16
3000
5200
2240
4226
34
19
26
16
18146
39133
106812
4226
:nEliminatio Forward theofSummary
4
3
2
1
4
3
2
1
x
x
x
x
x
x
x
x
CISE301_Topic3 KFUPM 20
ExampleBackward Substitution
36
)1(4)2(2)1(216,1
4
)1(2)2(26
22
59,1
3
3
for solve ,...for solvethen ,for Solve
3
9
6
16
3000
5200
2240
4226
12
34
134
4
3
2
1
xx
xx
xxx
x
x
x
x
CISE301_Topic3 KFUPM 21
Forward Elimination
ni
ba
abb
njaa
aaa
x
ni
ba
abb
njaa
aaa
x
ijj
ji
ijij
ijj
ji
ijij
3
)2(
eliminate To
2
)1(
eliminate To
222
2
222
2
2
111
1
111
1
1
CISE301_Topic3 KFUPM 22
Forward Elimination
.eliminated is until Continue
1
)(
eliminate To
1
n
mmm
imjj
mjmm
imijij
m
x
nim
ba
abb
njmaa
aaa
x
CISE301_Topic3 KFUPM 23
Backward Substitution
mm
n
mjjjmm
m
nn
nnnnnnnn
nn
nnnnn
nn
nn
a
xab
x
a
xaxabx
a
xabx
a
bx
,
1,
2,2
11,2,222
1,1
,111
CISE301_Topic3 KFUPM 24
Summary of the Naive Gaussian Elimination Example How to check a solution Problems with Naive Gaussian Elimination
Failure due to zero pivot element Error
Lecture 14Naive Gaussian
Elimination
CISE301_Topic3 KFUPM 25
Naive Gaussian Elimination
o The method consists of two stepso Forward Elimination: the system is reduced to
upper triangular form. A sequence of elementary operations is used.
o Backward Substitution: Solve the system starting from the last variable. Solve for xn ,xn-1,…x1.
'
'
'00
''0
3
2
1
3
2
1
33
2322
131211
3
2
1
3
2
1
333231
232221
131211
b
b
b
x
x
x
a
aa
aaa
b
b
b
x
x
x
aaa
aaa
aaa
CISE301_Topic3 KFUPM 26
Example 1
1775
64
832
11
333723
11
22210232
)(1832
3 ,2equations from Eliminate:Step1___nEliminatio Forward:1Part
:nEliminatioGaussian Naive using Solve
32
32
321
321
321
321
1
xx
xx
xxx
eqeqeqxxx
eqeqeqxxx
equationpivotunchangedeqxxx
x
CISE301_Topic3 KFUPM 27
Example 1
1313
64
832
21
5331775
)(264
1832
3equationfrom Eliminate:Step2nEliminatio Forward:1Part
3
32
321
32
32
321
2
x
xx
xxx
eqeqeqxx
equationpivotunchangedeqxx
unchangedeqxxx
x
CISE301_Topic3 KFUPM 28
Example 1Backward Substitution
1
2
1
issolution The
1328
21
46
113
13
3
2
1
1,1
32
1,1
33,122,111
3
2,2
33,222
3,3
33
x
x
x
a
xx
a
xaxabx
x
a
xabx
a
bx
CISE301_Topic3 KFUPM 29
How Do We Know If a Solution is Good or Not Given AX=B
X is a solution if AX-B=0 Due to computation error AX-B may not be zero Compute the residuals R=|AX-B|
One possible test is ?????
ii
rmax if acceptable issolution The
CISE301_Topic3 KFUPM 30
Determinant
13detdet
1300
410
321
A'
213
232
321
A
:Example
tdeterminan affect thenot do operations elementary The
operations Elementary
(A')(A)
CISE301_Topic3 KFUPM 31
How Many Solutions Does a System of Equations AX=B Have?
0 elements0 elements
B ingcorrespondB ingcorrespond
rows zerorows zero
moreor one has moreor one hasrows zero no has
matrix reducedmatrix reducedmatrix reduced
0det(A)0det(A)0det(A)
Infinitesolution NoUnique
CISE301_Topic3 KFUPM 32
Examples
5.1
!105.0
0
#:
0
2
00
21
1
2
00
21
1
1
20
21
4
2
42
21
3
2
42
21
2
1
43
21
solutions of # infintesolution NoUnique
XimpossibleX
solutionsInfinitesolutionNosolution
XXX
XXX
CISE301_Topic3 KFUPM 33
Lectures 15-16:Gaussian Elimination
with Scaled Partial Pivoting
Problems with Naive Gaussian Elimination Definitions and Initial step Forward Elimination Backward substitution Example
CISE301_Topic3 KFUPM 34
Problems with Naive Gaussian Elimination
o The Naive Gaussian Elimination may fail for very simple cases. (The pivoting element is zero).
o Very small pivoting element may result in serious computation errors
2
1
11
10
2
1
x
x
2
1
11
110
2
110
x
x
CISE301_Topic3 KFUPM 35
Example 2
1
1
1
1
3524
3685
4123
1211
:Pivoting Partial Scaledwith
nEliminatioGaussian using sytstem following theSolve
4
3
2
1
x
x
x
x
CISE301_Topic3 KFUPM 36
Example 2Initialization step
4321LVectorIndex
5842S vectorScale
1
1
1
1
3524
3685
4123
1211
4
3
2
1
x
x
x
xScale vector:
disregard sign
find largest in magnitude in each row
CISE301_Topic3 KFUPM 37
Why Index Vector? Index vectors are used because it is much
easier to exchange a single index element compared to exchanging the values of a complete row.
In practical problems with very large N, exchanging the contents of rows may not be practical since they could be stored at different locations.
CISE301_Topic3 KFUPM 38
Example 2Forward Elimination-- Step 1: eliminate x1
]1324[
Exchangeequation pivot first theis 4equation
toscorrespondmax 5
4,
8
5,
4
3,
2
14,3,2,1
]4321[
]5842[
1
1
1
1
3524
3685
4123
1211
equationpivot theofSelection
14
4
1,
4
3
2
1
L
landl
liS
aRatios
L
S
x
x
x
x
i
i
l
l
CISE301_Topic3 KFUPM 39
Example 2Forward Elimination-- Step 1: eliminate x1
1
25.2
75.1
25.1
3524
75.025.05.50
75.175.25.00
25.075.05.10
1
1
1
1
3524
3685
4123
1211
B andA Update
4
3
2
1
4
3
2
1
x
x
x
x
x
x
x
x
First pivot equation
CISE301_Topic3 KFUPM 40
Example 2Forward Elimination-- Step 2: eliminate x2
]2314[
2
5.1
8
5.5
4
5.0:Ratios
1
25.2
75.1
25.1
3524
75.025.05.50
75.175.25.00
25.075.05.10
equationpivot second theofSelection
4
3
2
1
L
x
x
x
x
CISE301_Topic3 KFUPM 41
Example 2Forward Elimination-- Step 2: eliminate x2
]2314[2
5.1
8
5.5
4
5.04,3,2:Ratios
]1324[]5842[
1
25.2
75.1
25.1
3524
75.025.05.50
75.175.25.00
25.075.05.10
equationpivot second theofSelection
2,
4
3
2
1
LiS
a
LS
x
x
x
x
i
i
l
l
CISE301_Topic3 KFUPM 42
Example 2Forward Elimination-- Step 3: eliminate x3
1
9
1667.2
25.1
3524
2000
8333.15.200
25.075.05.10
]3214[
1
8333.6
1667.2
25.1
3524
6667.125.000
8333.15.200
25.075.05.10
4
3
2
1
4
3
2
1
x
x
x
x
L
x
x
x
xThird pivot equation
CISE301_Topic3 KFUPM 43
Example 2Backward Substitution
7.23334
2531
1.13335.1
75.025.025.1
2.43275.2
8333.11667.2,5.4
2
9
]3214[
1
9
1667.2
25.1
3524
2000
8333.15.200
25.075.05.10
234
1,
22,33,44,1
34
2,
33,44,2
4
3,
44,3
4,4
4
3
2
1
1
1111
2
222
3
33
4
4
xxx
a
xaxaxabx
xx
a
xaxabx
x
a
xabx
a
bx
L
x
x
x
x
l
llll
l
lll
l
ll
l
l
CISE301_Topic3 KFUPM 44
Example 3
1
1
1
1
3524
3685
4123
1211
Pivoting Partial Scaledwith
nEliminatioGaussian using sytstem following theSolve
4
3
2
1
x
x
x
x
CISE301_Topic3 KFUPM 45
Example 3Initialization step
4321LVectorIndex
5842S vectorScale
1
1
1
1
3524
3685
4123
1211
4
3
2
1
x
x
x
x
CISE301_Topic3 KFUPM 46
Example 3Forward Elimination-- Step 1: eliminate x1
]1324[
Exchangeequation pivot first theis 4equation
toscorrespondmax 5
4,
8
5,
4
3,
2
14,3,2,1
]4321[
]5842[
1
1
1
1
3524
3685
4123
1211
equationpivot theofSelection
14
41,
4
3
2
1
L
landl
liS
aRatios
L
S
x
x
x
x
i
i
l
l
CISE301_Topic3 KFUPM 47
Example 3Forward Elimination-- Step 1: eliminate x1
1
25.2
75.1
25.1
3524
75.025.05.100
75.175.25.00
25.075.05.10
1
1
1
1
3524
3685
4133
1211
B andA Update
4
3
2
1
4
3
2
1
x
x
x
x
x
x
x
x
CISE301_Topic3 KFUPM 48
Example 3Forward Elimination-- Step 2: eliminate x2
]1234[
2
5.1
8
5.10
4
5.0
2
5.1
8
5.10
4
5.0:Ratios
1
25.2
75.1
25.1
3524
75.025.05.100
75.175.25.00
25.075.05.10
equationpivot second theofSelection
4
3
2
1
L
x
x
x
x
CISE301_Topic3 KFUPM 49
Example 3Forward Elimination-- Step 2: eliminate x2
]1234[2
5.1
8
5.10
4
5.04,3,2:Ratios
]1324[]5842[
1
25.2
75.1
25.1
3524
75.025.05.100
75.175.25.00
25.075.05.10
equationpivot second theofSelection
2,
4
3
2
1
LiS
a
LS
x
x
x
x
i
i
l
l
CISE301_Topic3 KFUPM 50
Example 3Forward Elimination-- Step 2: eliminate x2
1
2.25
1.8571
0.9286
3524
75.025.05.100
1.71432.7619-00
0.35710.785700
]2314[
1
25.2
75.1
25.1
3524
75.025.05.100
75.175.25.00
25.075.05.10
B andA Updating
4
3
2
1
4
3
2
1
x
x
x
x
L
x
x
x
x
CISE301_Topic3 KFUPM 51
Example 3Forward Elimination-- Step 3: eliminate x3
]1234[2
0.7857
4
2.76194,3:Ratios
]1234[]5842[
1
2.25
1.8571
0.9286
3524
75.025.05.100
1.71432.761900
0.35710.785700
equationpivot third theofSelection
3,
4
3
2
1
LiS
a
LS
x
x
x
x
i
i
l
l
CISE301_Topic3 KFUPM 52
Example 3Forward Elimination-- Step 3: eliminate x3
1
2.25
1.8571
1.4569
3524
75.025.05.100
1.71432.761900
0.8448000
]1234[
1
2.25
1.8571
0.9286
3524
75.025.05.100
1.71432.761900
0.35710.785700
4
3
2
1
4
3
2
1
x
x
x
x
L
x
x
x
x
CISE301_Topic3 KFUPM 53
Example 3Backward Substitution
1.86734
2531
0.3469
0.39802.7619
1.71431.8571,1.7245
0.8448
1.4569
]1234[
1
2.25
1.8571
1.4569
3524
75.025.05.100
1.71432.761900
0.8448000
234
1,
22,33,44,1
2,
33,44,2
4
3,
44,3
4,4
4
3
2
1
1
1111
2
222
3
33
4
4
xxx
a
xaxaxabx
a
xaxabx
x
a
xabx
a
bx
L
x
x
x
x
l
llll
l
lll
l
ll
l
l
CISE301_Topic3 KFUPM 54
How Good is the Solution?
0.001
0.003
0.002
0.005
:Residues
1.7245
0.3980
0.3469
1.8673
1
1
1
1
3524
3685
4123
1211
4
3
2
1
4
3
2
1
R
x
x
x
x
solution
x
x
x
x
CISE301_Topic3 KFUPM 55
Remarks: We use index vector to avoid the need to move
the rows which may not be practical for large problems.
If we order the equation as in the last value of the index vector, we have a triangular form.
Scale vector is formed by taking maximum in magnitude in each row.
Scale vector do not change. The original matrices A and B are used in
checking the residuals.
CISE301_Topic3 KFUPM 56
Lecture 17 Tridiagonal & Banded
Systems and Gauss-Jordan
Method
Tridiagonal Systems Diagonal Dominance Tridiagonal Algorithm Examples Gauss-Jordan Algorithm
CISE301_Topic3 KFUPM 57
Tridiagonal Systems: The non-zero elements are
in the main diagonal, super diagonal and subdiagonal.
aij=0 if |i-j| > 1
5
4
3
2
1
5
4
3
2
1
61000
14100
02620
00143
00015
b
b
b
b
b
x
x
x
x
x
Tridiagonal Systems
CISE301_Topic3 KFUPM 58
Occur in many applications Needs less storage (4n-2 compared to n2 +n for the general cases)
Selection of pivoting rows is unnecessary (under some conditions)
Efficiently solved by Gaussian elimination
Tridiagonal Systems
CISE301_Topic3 KFUPM 59
Based on Naive Gaussian elimination. As in previous Gaussian elimination algorithms
Forward elimination step Backward substitution step
Elements in the super diagonal are not affected. Elements in the main diagonal, and B need
updating
Algorithm to Solve Tridiagonal Systems
CISE301_Topic3 KFUPM 60
Tridiagonal System
nnn
n
nnnn
n
b
b
b
b
x
x
x
x
d
c
d
cd
cd
b
b
b
b
x
x
x
x
da
c
da
cda
cd
b a, d
3
2
1
3
2
1
1
3
22
11
3
2
1
3
2
1
1
1
32
221
11
and and d update toneed We
CISE301_Topic3 KFUPM 61
Diagonal Dominance
row. ingcorrespond in the elements of sum the
nlarger tha iselement diagonaleach of magnitude The
)1(aa
ifdominant diagonally is matrix A
,1ijii nifor
An
ijj
CISE301_Topic3 KFUPM 62
Diagonal Dominance
dominant DiagonallyNot dominant Diagonally
121
232
103
521
161
103
:Examples
CISE301_Topic3 KFUPM 63
Diagonally Dominant Tridiagonal System
A tridiagonal system is diagonally dominant if
Forward Elimination preserves diagonal dominance
)1(1 niacd iii
CISE301_Topic3 KFUPM 64
Solving Tridiagonal System
1,...,2,1for 1
onSubstituti Backward
2
nEliminatio Forward
1
11
1
11
1
nnixcbd
x
d
bx
nibd
abb
cd
add
iiii
i
n
nn
ii
iii
ii
iii
CISE301_Topic3 KFUPM 65
Example
1,2,3for 1
,
onSubstituti Backward
42,
nEliminatio Forward
6
8
9
12
,2
2
2
,1
1
1
,
5
5
5
5
6
8
9
12
51
251
251
25
Solve
1
11
11
1
1
4
3
2
1
ixcbd
xd
bx
ibd
abbc
d
add
BCAD
x
x
x
x
iiii
in
nn
ii
iiii
i
iii
CISE301_Topic3 KFUPM 66
Example
5619.45652.4
5652.616,5619.4
5652.4
215
5652.66.4
6.618,5652.4
6.4
215
6.65
1219,6.4
5
215
nEliminatio Forward
6
8
9
12
,2
2
2
,1
1
1
,
5
5
5
5
33
3443
3
344
22
2332
2
233
11
1221
1
122
bd
abbc
d
add
bd
abbc
d
add
bd
abbc
d
add
BCAD
CISE301_Topic3 KFUPM 67
ExampleBackward Substitution
After the Forward Elimination:
Backward Substitution:
25
1212
16.4
126.6
15652.4
125652.6
,15619.4
5619.4
5619.45652.66.612,5619.45652.46.45
1
2111
2
3222
3
4333
4
44
d
xcbx
d
xcbx
d
xcbx
d
bx
BD TT
CISE301_Topic3 KFUPM 68
Gauss-Jordan Method The method reduces the general system of
equations AX=B to IX=B where I is an identity matrix.
Only Forward elimination is done and no substitution is needed.
It has the same problems as Naive Gaussian elimination and can be modified to do partial scaled pivoting.
It takes 50% more time than Naive Gaussian method.
CISE301_Topic3 KFUPM 69
Gauss-Jordan MethodExample
2
7
0
200
560
111
11
233
11
422
2/11
32fromxEleminate1
2
7
0
422
124
222
3
2
1
1
3
2
1
x
x
x
eqeqeq
eqeqeq
eqeq
andequationsStep
x
x
x
CISE301_Topic3 KFUPM 70
Gauss-Jordan MethodExample
2
1.1667
1.1667
200
0.833310
1667.001
21
033
21
111
6/22
31fromxEleminate2
2
7
0
200
560
111
3
2
1
2
3
2
1
x
x
x
eqeqeq
eqeqeq
eqeq
andequationsStep
x
x
x
CISE301_Topic3 KFUPM 71
Gauss-Jordan MethodExample
1
2
1
100
010
001
31
0.833322
31
1667.011
2/33
21fromxEleminate3
2
1.1667
1.1667
200
0.833310
1667.001
3
2
1
3
3
2
1
x
x
x
eqeqeq
eqeqeq
eqeq
andequationsStep
x
x
x
CISE301_Topic3 KFUPM 72
Gauss-Jordan MethodExample
1
2
1
1
2
1
100
010
001
2
7
0
422
124
222
3
2
1
3
2
1
3
2
1
x
x
x
issolution
x
x
x
todtransformeis
x
x
x