lecture 6 matrix operations and gaussian elimination for solving linear systems shang-hua teng
TRANSCRIPT
Lecture 6Matrix Operations
and Gaussian Elimination for Solving Linear Systems
Shang-Hua Teng
Matrix (Uniform Representation for Any Dimension)
• An m by n matrix is a rectangular table of mn numbers
ji
nmmm
n
n
ajiA
aaa
aaa
aaa
A
,
,2,1,
,22,21,2
,12,11,1
),( write weSometime
...
...
...
...
Matrix (Uniform Representation for Any Dimension)
• Can be viewed as m row vectors in n dimensions
nmmm
n
n
aaa
aaa
aaa
A
,2,1,
,22,21,2
,12,11,1
...
...
...
...
Matrix (Uniform Representation for Any Dimension)
• Or can be viewed as n column vectors in m dimensions
nmmm
n
n
aaa
aaa
aaa
A
,2,1,
,22,21,2
,12,11,1
...
...
...
...
Squared Matrix
• An n by n matrix is a squared table of n2 numbers
nnnn
n
n
aaa
aaa
aaa
A
,2,1,
,22,21,2
,12,11,1
...
...
...
...
Some Special Squared Matrices
• All zeros matrix
0...00
...
0...00
0...00
),(0
mn
• Identity matrix
1...00
...
0...10
0...01
),(
nnII
Matrix Operations
• Addition
• Scalar multiplication
• Multiplication
1. Matrix Addition:
mnmnmm
nn
nn
mnmm
n
n
mnmm
n
n
baba
bababa
bababa
BA
bbb
bbb
bbb
B
a aa
a aa
a aa
A
,
,
11
2222222121
1112121111
21
22221
11211
21
22221
11211
Matrices have to have the same dimensionsWhat is the complexity?
2. Scalar Multiplication:
mnmm
n
n
mnmm
n
n
aaa
aaa
aaa
aaa
aaa
aaa
A
21
2 2221
11211
21
22221
11211
What is the complexity?
3. Matrix Multiplication
n
i ipmiimi
n
i ipi
n
i ii
n
i ii
n
i ipi
n
i ii
n
i ii
npnn
p
p
mnmm
n
n
baba
bababa
bababa
BA
bbb
bbb
bbb
B
aaa
aaa
aaa
A
1
n
1=i 1
1 21 221 12
1 11 211 11
21
22221
11211
21
22221
11211
,
,
Two matrices have to be conformalWhat is the complexity?
Matrix Multiplication
B) of j(column A) of i row(
,
,
21
22221
11211
21
22221
11211
BA
bbb
bbb
bbb
B
aaa
aaa
aaa
A
npnn
p
p
mnmm
n
n
Two matrices have to be conformal
The Laws of Matrix Operations
• A + B = B + A (commutative)
• c(A+B) = cA + c+B (distributive)
• A + (B + C) = (A + B) + C (associative)
• C(A+B) = CA + CB (distributive from left)
• (A+B)C = AC+BC (distributive from right)
• A(BC) = (AB)C (associative)
• But in general: BAAB
Counter Example
00
01
01
00
00
10
10
00
00
10
01
00
BA
but
AB
Special Matrices
• Identity matrix I– IA = AI = A
• Square Matrix A
pqqp
qpqp
p
p
AA
AAA
AAAAA
Elimination: Method for Solving Linear Systems
• Linear Systems == System of Linear Equations
• Elimination: – Multiply the LHS and RHS of an equation by a
nonzero constant results the same equations
– Adding the LHSs and RHSs of two equations does not change the solution
0),()(:)()(: xgxfxxgxfx
)()()()();()(:)()();()(: 2121112211 xgxgxfxfxgxfxxgxfxgxfx
Elimination in 2D
• Multiply the first equation by 3 and subtracts from the second equation (to eliminate x)
1123
12
yx
yx
880
12
y
yx
• The two systems have the same solution
• The second system is easy to solve
Geometry of Elimination
1123
12
yx
yx
1123 yx
12 yx
(3,1)8y = 8
880
12
y
yxReduce to a 1-dimensional problem.
Upper Triangular Systems and Back Substitution
• Back substitution– From the second equation y = 1– Substitute the value of y to the first equation to obtain
x-2=1– Solve it we have: x = 3
880
12
y
yx
• So the solution is (3,1)
How Much to Multiply before Subtracting
• Pivot: first nonzero in the row that does the elimination
• Multiplier: (entry to eliminate) divided by (pivot)
1123
12
yx
yx
Multiply: = 3/1
How Much to Multiply before Subtracting
• Pivot: first nonzero in the row that does the elimination
• Multiplier: (entry to eliminate) divided by (pivot)
1123
242
yx
yx
Multiply: = 3/2
880
242
y
yx
The pivots are on the diagonal of the triangle after the elimination
Breakdown of Elimination
• What is the pivot is zero == one can’t divide by zero!!!!
1163
12
yx
yx
Eliminate x:
80
12
y
yx
No Solution!!!!: this system has no second pivot
Geometric Intuition(Row Pictures)
• Two parallel lines never intersect
1163 yx
12 yx
(3,1)8y = 8
Geometric Intuition(Column Picture)
11
1
3
1
6
2
Two column vectors are co-linear!!!!
1163
12
yx
yx
Geometric Intuition
Geometric degeneracy cause failure in elimination!
Failure in Elimination May Indicate Infinitely Many Solutions
• y is free, can be number!
• Geometric Intuition (row picture): The two line are the same
• Geometric Intuition (column picture): all three column vectors are co-linear
363
12
yx
yx00
12
y
yx
Failure in Elimination(Temporary and can be Fixed)
• First pivot position contains zero• Exchange with the second equation
523
420
yx
yx
42
522
y
yx
Can be solved by backward substitution!
Singular Systems versus Non-Singular Systems
• A singular system has no solution or infinitely many solution– Row Picture: two line are parallel or the same– Column Picture: Two column vectors are co-
linear
• A non-singular system has a unique solution– Row Picture: two non-parallel lines– Column Picture: two non-colinear column
vectors
Gaussian Elimination in 3D
• Using the first pivot to eliminate x from the next two equations
10732
8394
2242
zyx
zyx
zyx
Gaussian Elimination in 3D
• Using the second pivot to eliminate y from the third equation
125
4
2242
zy
zy
zyx
Gaussian Elimination in 3D
• Using the second pivot to eliminate y from the third equation
84
4
2242
z
zy
zyx
Now We Have a Triangular System
• From the last equation, we have
84
4
2242
z
zy
zyx
Backward Substitution
• And substitute z to the first two equations
2
4
2242
z
zy
zyx
Backward Substitution
• We can solve y
2
42
2442
z
y
yx
Backward Substitution
• Substitute to the first equation
2
2
2442
z
y
yx
Backward Substitution
• We can solve the first equation
2
2
2482
z
y
x
Backward Substitution
• We can solve the first equation
2
2
1
z
y
x
Generalization
• How to generalize to higher dimensions?
• What is the complexity of the algorithm?
• Answer:
• Express Elimination with Matrices
Step 1Build Augmented Matrix
10732
8394
2242
zyx
zyx
zyx
Ax = b
10732
8394
2242
bA[A b]
Pivot 1: The elimination of column 1
1
2
10732
8394
2242
10732
4110
2242
12510
4110
2242
Pivot 2: The elimination of column 2
1
12510
4110
2242
8400
4110
2242
Upper triangular matrix
Backward Substitution 1: from the last column to the first
8400
4110
2242
Upper triangular matrix
2100
4110
2242
2100
2010
2242
2100
2010
6042
2100
2010
2002
2100
2010
1001
Expressing Elimination by Matrix Multiplication
Elementary or Elimination Matrix
• The elementary or elimination matrix
That subtracts a multiple l of row j from row i can be obtained from the identity entry by adding (-l) in the i,j position
jiE ,
jiE ,
10
010
001
1,3
l
E
Elementary or Elimination Matrix
3,33,12,32,11,31,1
3,22,21,2
3,12,11,1
3,32,31,3
3,22,21,2
3,12,11,1
3,32,31,3
3,22,21,2
3,12,11,1
1,3
10
010
001
alaalaala
aaa
aaa
aaa
aaa
aaa
laaa
aaa
aaa
E
Pivot 1: The elimination of column 1
12510
4110
2242
1
2
Elimination matrix
10732
8394
2242
10732
4110
2242
10732
8394
2242
100
012
001
12510
4110
2242
10732
4110
2242
101
010
001
The Product of Elimination Matrices
101
012
001
100
012
001
101
010
001
111
012
001
101
012
001
110
010
001