lu factorization2

132
Direct Methods for Solving Linear Systems Matrix Factorization Numerical Analysis (9th Edition) R L Burden & J D Faires Beamer Presentation Slides prepared by John Carroll Dublin City University c 2011 Brooks/Cole, Cengage Learning

Upload: kkamal600

Post on 08-Apr-2016

11 views

Category:

Documents


0 download

DESCRIPTION

lu factorization2

TRANSCRIPT

Page 1: Lu Factorization2

Direct Methods for Solving Linear Systems

Matrix Factorization

Numerical Analysis (9th Edition)

R L Burden & J D Faires

Beamer Presentation Slidesprepared byJohn Carroll

Dublin City University

c© 2011 Brooks/Cole, Cengage Learning

Page 2: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Outline

1 Computation Cost Rationale & Basic Solution Strategy

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 2 / 46

Page 3: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Outline

1 Computation Cost Rationale & Basic Solution Strategy

2 Constructing the Matrix Factorization

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 2 / 46

Page 4: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Outline

1 Computation Cost Rationale & Basic Solution Strategy

2 Constructing the Matrix Factorization

3 Example: LU Factorization of a 4 × 4 Matrix

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 2 / 46

Page 5: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Outline

1 Computation Cost Rationale & Basic Solution Strategy

2 Constructing the Matrix Factorization

3 Example: LU Factorization of a 4 × 4 Matrix

4 The LU Factorization Algorithm

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 2 / 46

Page 6: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Outline

1 Computation Cost Rationale & Basic Solution Strategy

2 Constructing the Matrix Factorization

3 Example: LU Factorization of a 4 × 4 Matrix

4 The LU Factorization Algorithm

5 Permutation Matrices for Row Interchanges

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 2 / 46

Page 7: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Outline

1 Computation Cost Rationale & Basic Solution Strategy

2 Constructing the Matrix Factorization

3 Example: LU Factorization of a 4 × 4 Matrix

4 The LU Factorization Algorithm

5 Permutation Matrices for Row Interchanges

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 3 / 46

Page 8: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Background

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 4 / 46

Page 9: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

BackgroundGaussian elimination is the principal tool in the direct solution oflinear systems of equations.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 4 / 46

Page 10: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

BackgroundGaussian elimination is the principal tool in the direct solution oflinear systems of equations.

We will now see that the steps used to solve a system of the formAx = b can be used to factor a matrix.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 4 / 46

Page 11: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

BackgroundGaussian elimination is the principal tool in the direct solution oflinear systems of equations.

We will now see that the steps used to solve a system of the formAx = b can be used to factor a matrix.

The factorization is particularly useful when it has the formA = LU, where L is lower triangular and U is upper triangular.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 4 / 46

Page 12: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

BackgroundGaussian elimination is the principal tool in the direct solution oflinear systems of equations.

We will now see that the steps used to solve a system of the formAx = b can be used to factor a matrix.

The factorization is particularly useful when it has the formA = LU, where L is lower triangular and U is upper triangular.

Although not all matrices have this type of representation, manydo that occur frequently in the application of numerical techniques.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 4 / 46

Page 13: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Computational Cost ConsiderationsGaussian elimination applied to an arbitrary linear system Ax = brequires O(n3/3) arithmetic operations to determine x.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 5 / 46

Page 14: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Computational Cost ConsiderationsGaussian elimination applied to an arbitrary linear system Ax = brequires O(n3/3) arithmetic operations to determine x.

However, to solve a linear system that involves an upper-triangularsystem requires only backward substitution, which takes O(n2)operations.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 5 / 46

Page 15: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Computational Cost ConsiderationsGaussian elimination applied to an arbitrary linear system Ax = brequires O(n3/3) arithmetic operations to determine x.

However, to solve a linear system that involves an upper-triangularsystem requires only backward substitution, which takes O(n2)operations.

The number of operations required to solve a lower-triangularsystems is similar.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 5 / 46

Page 16: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Solution StrategySuppose that A has been factored into the triangular form A = LU,where L is lower triangular and U is upper triangular.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 6 / 46

Page 17: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Solution StrategySuppose that A has been factored into the triangular form A = LU,where L is lower triangular and U is upper triangular. Then we cansolve for x more easily by using a two-step process:

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 6 / 46

Page 18: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Solution StrategySuppose that A has been factored into the triangular form A = LU,where L is lower triangular and U is upper triangular. Then we cansolve for x more easily by using a two-step process:

First we let y = Ux and solve the lower triangular system Ly = bfor y. Since L is triangular, determining y from this equationrequires only O(n2) operations.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 6 / 46

Page 19: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Solution StrategySuppose that A has been factored into the triangular form A = LU,where L is lower triangular and U is upper triangular. Then we cansolve for x more easily by using a two-step process:

First we let y = Ux and solve the lower triangular system Ly = bfor y. Since L is triangular, determining y from this equationrequires only O(n2) operations.

Once y is known, the upper triangular system Ux = y requiresonly an additional O(n2) operations to determine the solution x.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 6 / 46

Page 20: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Solution StrategySuppose that A has been factored into the triangular form A = LU,where L is lower triangular and U is upper triangular. Then we cansolve for x more easily by using a two-step process:

First we let y = Ux and solve the lower triangular system Ly = bfor y. Since L is triangular, determining y from this equationrequires only O(n2) operations.

Once y is known, the upper triangular system Ux = y requiresonly an additional O(n2) operations to determine the solution x.

Solving a linear system Ax = b in factored form means that thenumber of operations needed to solve the system Ax = b is reducedfrom O(n3/3) to O(2n2).

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 6 / 46

Page 21: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Outline

1 Computation Cost Rationale & Basic Solution Strategy

2 Constructing the Matrix Factorization

3 Example: LU Factorization of a 4 × 4 Matrix

4 The LU Factorization Algorithm

5 Permutation Matrices for Row Interchanges

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 7 / 46

Page 22: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & UFirst, suppose that Gaussian elimination can be performed on thesystem Ax = b without row interchanges.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 8 / 46

Page 23: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & UFirst, suppose that Gaussian elimination can be performed on thesystem Ax = b without row interchanges.

With the notation used earlier, this is equivalent to having nonzeropivot elements a(i)

ii , for each i = 1, 2, . . . , n.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 8 / 46

Page 24: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & UFirst, suppose that Gaussian elimination can be performed on thesystem Ax = b without row interchanges.

With the notation used earlier, this is equivalent to having nonzeropivot elements a(i)

ii , for each i = 1, 2, . . . , n.

The first step in the Gaussian elimination process consists ofperforming, for each j = 2, 3, . . . , n, the operations

(Ej − mj ,1E1) → (Ej), where mj ,1 =a(1)

j1

a(1)11

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 8 / 46

Page 25: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & UFirst, suppose that Gaussian elimination can be performed on thesystem Ax = b without row interchanges.

With the notation used earlier, this is equivalent to having nonzeropivot elements a(i)

ii , for each i = 1, 2, . . . , n.

The first step in the Gaussian elimination process consists ofperforming, for each j = 2, 3, . . . , n, the operations

(Ej − mj ,1E1) → (Ej), where mj ,1 =a(1)

j1

a(1)11

These operations transform the system into one in which all theentries in the first column below the diagonal are zero.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 8 / 46

Page 26: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Constructing L & U (Cont’d)

The system of operations in

(Ej − mj ,1E1) → (Ej), where mj ,1 =a(1)

j1

a(1)11

can be viewed in another way.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 9 / 46

Page 27: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Constructing L & U (Cont’d)

The system of operations in

(Ej − mj ,1E1) → (Ej), where mj ,1 =a(1)

j1

a(1)11

can be viewed in another way. It is simultaneously accomplished bymultiplying the original matrix A on the left by the matrix

M(1) =

1 0 · · · · · · 0

−m21 1. . .

...... 0

. . . . . ....

......

. . . . . . 0−mn1 0 · · · 0 1

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 9 / 46

Page 28: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Constructing L & U (Cont’d)

The system of operations in

(Ej − mj ,1E1) → (Ej), where mj ,1 =a(1)

j1

a(1)11

can be viewed in another way. It is simultaneously accomplished bymultiplying the original matrix A on the left by the matrix

M(1) =

1 0 · · · · · · 0

−m21 1. . .

...... 0

. . . . . ....

......

. . . . . . 0−mn1 0 · · · 0 1

This is called the first Gaussian transformation matrix.Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 9 / 46

Page 29: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

We denote the product of this matrix with A(1) ≡ A by A(2) andwith b by b(2),

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 10 / 46

Page 30: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

We denote the product of this matrix with A(1) ≡ A by A(2) andwith b by b(2), so

A(2)x = M(1)Ax = M(1)b = b(2)

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 10 / 46

Page 31: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

We denote the product of this matrix with A(1) ≡ A by A(2) andwith b by b(2), so

A(2)x = M(1)Ax = M(1)b = b(2)

In a similar manner we construct M(2), the identity matrix with theentries below the diagonal in the second column replaced by thenegatives of the multipliers

mj ,2 =a(2)

j2

a(2)22

.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 10 / 46

Page 32: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

The product of M(2) with A(2) has zeros below the diagonal in thefirst two columns,

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 11 / 46

Page 33: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

The product of M(2) with A(2) has zeros below the diagonal in thefirst two columns, and we let

A(3)x = M(2)A(2)x = M(2)M(1)Ax = M(2)M(1)b = b(3)

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 11 / 46

Page 34: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

In general, with A(k)x = b(k) already formed,

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 12 / 46

Page 35: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

In general, with A(k)x = b(k) already formed, multiply by the k thGaussian transformation matrix

M(k) =

1 0 · · · · · · · · · · · · · · · 0

0 1. . .

......

. . . . . . . . ....

... 0. . . . . .

......

... −mk+1,k. . . . . .

......

......

. . . . . . . . . 0... · · · 0 −mn,k 0 · · · 0 1

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 12 / 46

Page 36: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)to obtain

A(k+1)x = M(k)A(k)x

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 13 / 46

Page 37: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)to obtain

A(k+1)x = M(k)A(k)x

= M(k) · · ·M(1)Ax

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 13 / 46

Page 38: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)to obtain

A(k+1)x = M(k)A(k)x

= M(k) · · ·M(1)Ax

= M(k)b(k)

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 13 / 46

Page 39: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)to obtain

A(k+1)x = M(k)A(k)x

= M(k) · · ·M(1)Ax

= M(k)b(k)

= b(k+1)

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 13 / 46

Page 40: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)to obtain

A(k+1)x = M(k)A(k)x

= M(k) · · ·M(1)Ax

= M(k)b(k)

= b(k+1)

= M(k) · · ·M(1)b

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 13 / 46

Page 41: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

The process ends with the formation of A(n)x = b(n), where A(n) is theupper triangular matrix

A(n) =

a(1)11 a(1)

12 · · · · · · a(1)1n

0 a(2)22

. . ....

.... . . . . . . . .

......

. . . . . . a(n−1)n−1,n

0 · · · · · · 0 a(n)n,n

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 14 / 46

Page 42: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

The process ends with the formation of A(n)x = b(n), where A(n) is theupper triangular matrix

A(n) =

a(1)11 a(1)

12 · · · · · · a(1)1n

0 a(2)22

. . ....

.... . . . . . . . .

......

. . . . . . a(n−1)n−1,n

0 · · · · · · 0 a(n)n,n

given byA(n) = M(n−1)M(n−2) · · ·M(1)A

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 14 / 46

Page 43: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

This process forms the U = A(n) portion of the matrix factorizationA = LU.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 15 / 46

Page 44: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

This process forms the U = A(n) portion of the matrix factorizationA = LU.

To determine the complementary lower triangular matrix L,

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 15 / 46

Page 45: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

This process forms the U = A(n) portion of the matrix factorizationA = LU.

To determine the complementary lower triangular matrix L, firstrecall the multiplication of A(k)x = b(k) by the Gaussiantransformation of M(k) used to obtain:

A(k+1)x = M(k)A(k)x = M(k)b(k) = b(k+1),

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 15 / 46

Page 46: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

This process forms the U = A(n) portion of the matrix factorizationA = LU.

To determine the complementary lower triangular matrix L, firstrecall the multiplication of A(k)x = b(k) by the Gaussiantransformation of M(k) used to obtain:

A(k+1)x = M(k)A(k)x = M(k)b(k) = b(k+1),

where M(k) generates the row operations

(Ej − mj ,kEk ) → (Ej), for j = k + 1, . . . , n.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 15 / 46

Page 47: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

To reverse the effects of this transformation and return to A(k) requiresthat the operations (Ej + mj ,kEk) → (Ej) be performed for each

j = k + 1, . . . , n.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 16 / 46

Page 48: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

To reverse the effects of this transformation and return to A(k) requiresthat the operations (Ej + mj ,kEk) → (Ej) be performed for each

j = k + 1, . . . , n. This is equivalent to multiplying by[

M(k)]

−1:

L(k) =[

M(k)]

−1=

1 0 · · · · · · · · · · · · · · · 0

0 1. . .

......

. . . . . . . . ....

... 0. . . . . .

......

... mk+1,k. . . . . .

......

......

. . . . . . . . . 0... · · · 0 mn,k 0 · · · 0 1

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 16 / 46

Page 49: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)The lower-triangular matrix L in the factorization of A, then, is theproduct of the matrices L(k):

L = L(1)L(2) · · · L(n−1) =

1 0 · · · · · · 0

m21 1. . .

......

. . . . . . . . ....

......

. . . . . . 0mn1 · · · · · · mn,n−1 1

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 17 / 46

Page 50: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)The lower-triangular matrix L in the factorization of A, then, is theproduct of the matrices L(k):

L = L(1)L(2) · · · L(n−1) =

1 0 · · · · · · 0

m21 1. . .

......

. . . . . . . . ....

......

. . . . . . 0mn1 · · · · · · mn,n−1 1

since the product of L with the upper-triangular matrixU = M(n−1) · · ·M(2)M(1)A gives

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 17 / 46

Page 51: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

LU = L(1)L(2) · · · L(n−3)L(n−2)L(n−1)

· M(n−1)M(n−2)M(n−3) · · ·M(2)M(1) A

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 18 / 46

Page 52: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

LU = L(1)L(2) · · · L(n−3)L(n−2)L(n−1)

· M(n−1)M(n−2)M(n−3) · · ·M(2)M(1) A

= [M(1)]−1[M(2)]−1 · · · [M(n−2)]−1[M(n−1)]−1

· M(n−1)M(n−2) · · ·M(2)M(1) A

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 18 / 46

Page 53: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

LU = L(1)L(2) · · · L(n−3)L(n−2)L(n−1)

· M(n−1)M(n−2)M(n−3) · · ·M(2)M(1) A

= [M(1)]−1[M(2)]−1 · · · [M(n−2)]−1[M(n−1)]−1

· M(n−1)M(n−2) · · ·M(2)M(1) A

= A

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 18 / 46

Page 54: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Constructing L & U (Cont’d)

LU = L(1)L(2) · · · L(n−3)L(n−2)L(n−1)

· M(n−1)M(n−2)M(n−3) · · ·M(2)M(1) A

= [M(1)]−1[M(2)]−1 · · · [M(n−2)]−1[M(n−1)]−1

· M(n−1)M(n−2) · · ·M(2)M(1) A

= A

We now state a theorem which follows from these observations.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 18 / 46

Page 55: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

TheoremIf Gaussian elimination can be performed on the linear system Ax = bwithout row interchanges,

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 19 / 46

Page 56: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

TheoremIf Gaussian elimination can be performed on the linear system Ax = bwithout row interchanges, then the matrix A can be factored into theproduct of a lower-triangular matrix L and an upper-triangular matrix U,

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 19 / 46

Page 57: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

TheoremIf Gaussian elimination can be performed on the linear system Ax = bwithout row interchanges, then the matrix A can be factored into theproduct of a lower-triangular matrix L and an upper-triangular matrix U,that is, A = LU, where mji = a(i)

ji /a(i)ii ,

U =

a(1)11 a(1)

12 · · · · · · a(1)1n

0 a(2)22

. . ....

.... . .

. . .. . .

......

. . .. . . a(n−1)

n−1,n

0 · · · · · · 0 a(n)n,n

L =

1 0 · · · · · · 0

m21 1. . .

......

. . .. . .

. . ....

......

. . .. . . 0

mn1 · · · · · · mn,n−1 1

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 19 / 46

Page 58: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Outline

1 Computation Cost Rationale & Basic Solution Strategy

2 Constructing the Matrix Factorization

3 Example: LU Factorization of a 4 × 4 Matrix

4 The LU Factorization Algorithm

5 Permutation Matrices for Row Interchanges

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 20 / 46

Page 59: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Example(a) Determine the LU factorization for matrix A in the linear system

Ax = b, where

A =

1 1 0 32 1 −1 13 −1 −1 2

−1 2 3 −1

and b =

11

−34

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 21 / 46

Page 60: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Example(a) Determine the LU factorization for matrix A in the linear system

Ax = b, where

A =

1 1 0 32 1 −1 13 −1 −1 2

−1 2 3 −1

and b =

11

−34

(b) Then use the factorization to solve the system

x1 + x2 + 3x4 = 8

2x1 + x2 − x3 + x4 = 7

3x1 − x2 − x3 + 2x4 = 14

−x1 + 2x2 + 3x3 − x4 = −7

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 21 / 46

Page 61: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: 4 × 4 Example

Part (a) Solution (1/2)The original system was considered under Gaussian Elimination

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 22 / 46

Page 62: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: 4 × 4 Example

Part (a) Solution (1/2)The original system was considered under Gaussian Eliminationwhere we saw that the sequence of operations

(E2 − 2E1) → (E2) (E3 − 3E1) → (E3)(E4 − (−1)E1) → (E4) (E3 − 4E2) → (E3)(E4 − (−3)E2) → (E4)

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 22 / 46

Page 63: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: 4 × 4 Example

Part (a) Solution (1/2)The original system was considered under Gaussian Eliminationwhere we saw that the sequence of operations

(E2 − 2E1) → (E2) (E3 − 3E1) → (E3)(E4 − (−1)E1) → (E4) (E3 − 4E2) → (E3)(E4 − (−3)E2) → (E4)

converts the system to the triangular system

x1 + x2 + 3x4 = 4

− x2 − x3 − 5x4 = −7

3x3 + 13x4 = 13

− 13x4 = −13

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 22 / 46

Page 64: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: 4 × 4 Example

Part (a) Solution (2/2)The multipliers mij and the upper triangular matrix produce thefactorization

A =

1 1 0 32 1 −1 13 −1 −1 2

−1 2 3 −1

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 23 / 46

Page 65: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: 4 × 4 Example

Part (a) Solution (2/2)The multipliers mij and the upper triangular matrix produce thefactorization

A =

1 1 0 32 1 −1 13 −1 −1 2

−1 2 3 −1

=

1 0 0 02 1 0 03 4 1 0

−1 −3 0 1

1 1 0 30 −1 −1 −50 0 3 130 0 0 −13

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 23 / 46

Page 66: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: 4 × 4 Example

Part (a) Solution (2/2)The multipliers mij and the upper triangular matrix produce thefactorization

A =

1 1 0 32 1 −1 13 −1 −1 2

−1 2 3 −1

=

1 0 0 02 1 0 03 4 1 0

−1 −3 0 1

1 1 0 30 −1 −1 −50 0 3 130 0 0 −13

= LU

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 23 / 46

Page 67: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: 4 × 4 Example

Part (b) Solution (1/3)To solve

Ax = LUx =

1 0 0 02 1 0 03 4 1 0

−1 −3 0 1

1 1 0 30 −1 −1 −50 0 3 130 0 0 −13

x1

x2

x3

x4

=

87

14−7

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 24 / 46

Page 68: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: 4 × 4 Example

Part (b) Solution (1/3)To solve

Ax = LUx =

1 0 0 02 1 0 03 4 1 0

−1 −3 0 1

1 1 0 30 −1 −1 −50 0 3 130 0 0 −13

x1

x2

x3

x4

=

87

14−7

we first introduce the substitution y = Ux.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 24 / 46

Page 69: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: 4 × 4 Example

Part (b) Solution (1/3)To solve

Ax = LUx =

1 0 0 02 1 0 03 4 1 0

−1 −3 0 1

1 1 0 30 −1 −1 −50 0 3 130 0 0 −13

x1

x2

x3

x4

=

87

14−7

we first introduce the substitution y = Ux. Then b = L(Ux) = Ly.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 24 / 46

Page 70: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: 4 × 4 Example

Part (b) Solution (2/3)First, solve Ly = b (where y = Ux:

Ly =

1 0 0 02 1 0 03 4 1 0

−1 −3 0 1

y1

y2

y3

y4

=

87

14−7

.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 25 / 46

Page 71: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: 4 × 4 Example

Part (b) Solution (2/3)First, solve Ly = b (where y = Ux:

Ly =

1 0 0 02 1 0 03 4 1 0

−1 −3 0 1

y1

y2

y3

y4

=

87

14−7

.

This system is solved for y by a simple forward-substitution process:

y1 = 8

2y1 + y2 = 7 ⇒ y2 = 7 − 2y1 = −9

3y1 + 4y2 + y3 = 14 ⇒ y3 = 14 − 3y1 − 4y2 = 26

−y1 − 3y2 + y4 = −7 ⇒ y4 = −7 + y1 + 3y2 = −26

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 25 / 46

Page 72: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: 4 × 4 Example

Part (b) Solution (3/3)We then solve Ux = y for x, the solution of the original system;

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 26 / 46

Page 73: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: 4 × 4 Example

Part (b) Solution (3/3)We then solve Ux = y for x, the solution of the original system; that is,

1 1 0 30 −1 −1 −50 0 3 130 0 0 −13

x1

x2

x3

x4

=

8−926

−26

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 26 / 46

Page 74: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: 4 × 4 Example

Part (b) Solution (3/3)We then solve Ux = y for x, the solution of the original system; that is,

1 1 0 30 −1 −1 −50 0 3 130 0 0 −13

x1

x2

x3

x4

=

8−926

−26

Using backward substitution we obtain x4 = 2, x3 = 0, x2 = −1, x1 = 3.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 26 / 46

Page 75: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Outline

1 Computation Cost Rationale & Basic Solution Strategy

2 Constructing the Matrix Factorization

3 Example: LU Factorization of a 4 × 4 Matrix

4 The LU Factorization Algorithm

5 Permutation Matrices for Row Interchanges

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 27 / 46

Page 76: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

LU Factorization Algorithm (1/3)

To factor the n × n matrix A = [aij ] into the product of thelower-triangular matrix L = [lij ] and the upper-triangular matrixU = [uij ]; that is, A = LU, where the main diagonal of either L or Uconsists of all ones:

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 28 / 46

Page 77: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

LU Factorization Algorithm (1/3)

To factor the n × n matrix A = [aij ] into the product of thelower-triangular matrix L = [lij ] and the upper-triangular matrixU = [uij ]; that is, A = LU, where the main diagonal of either L or Uconsists of all ones:

INPUT dimension n; the entries aij , 1 ≤ i , j ≤ n of A;the diagonal l11 = · · · = lnn = 1 of L orthe diagonal u11 = · · · = unn = 1 of U.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 28 / 46

Page 78: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

LU Factorization Algorithm (1/3)

To factor the n × n matrix A = [aij ] into the product of thelower-triangular matrix L = [lij ] and the upper-triangular matrixU = [uij ]; that is, A = LU, where the main diagonal of either L or Uconsists of all ones:

INPUT dimension n; the entries aij , 1 ≤ i , j ≤ n of A;the diagonal l11 = · · · = lnn = 1 of L orthe diagonal u11 = · · · = unn = 1 of U.

OUTPUT the entries lij , 1 ≤ j ≤ i , 1 ≤ i ≤ n of L and the entries,uij , i ≤ j ≤ n, 1 ≤ i ≤ n of U.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 28 / 46

Page 79: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

LU Factorization Algorithm (2/3)

Step 1 Select l11 and u11 satisfying l11u11 = a11

If l11u11 = 0 then OUTPUT (‘Factorization impossible’)STOP

Step 2 For j = 2, . . . , n set u1j = a1j/l11 (First row of U)lj1 = aj1/u11 (First column of L)

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 29 / 46

Page 80: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

LU Factorization Algorithm (2/3)

Step 1 Select l11 and u11 satisfying l11u11 = a11

If l11u11 = 0 then OUTPUT (‘Factorization impossible’)STOP

Step 2 For j = 2, . . . , n set u1j = a1j/l11 (First row of U)lj1 = aj1/u11 (First column of L)

Step 3 For i = 2, . . . , n − 1 do Steps 4 and 5:

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 29 / 46

Page 81: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

LU Factorization Algorithm (2/3)

Step 1 Select l11 and u11 satisfying l11u11 = a11

If l11u11 = 0 then OUTPUT (‘Factorization impossible’)STOP

Step 2 For j = 2, . . . , n set u1j = a1j/l11 (First row of U)lj1 = aj1/u11 (First column of L)

Step 3 For i = 2, . . . , n − 1 do Steps 4 and 5:

Step 4 Select lii and uii satisfying liiuii = aii −∑i−1

k=1 likuki

If liiuii = 0 then OUTPUT (‘Factorization impossible’)STOP

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 29 / 46

Page 82: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

LU Factorization Algorithm (2/3)

Step 1 Select l11 and u11 satisfying l11u11 = a11

If l11u11 = 0 then OUTPUT (‘Factorization impossible’)STOP

Step 2 For j = 2, . . . , n set u1j = a1j/l11 (First row of U)lj1 = aj1/u11 (First column of L)

Step 3 For i = 2, . . . , n − 1 do Steps 4 and 5:

Step 4 Select lii and uii satisfying liiuii = aii −∑i−1

k=1 likuki

If liiuii = 0 then OUTPUT (‘Factorization impossible’)STOP

Step 5 For j = i + 1, . . . , n

set uij = 1lii

[

aij −∑i−1

k=1 likukj

]

(i th row of U)

lji = 1uii

[

aji −∑i−1

k=1 ljkuki

]

(i th column of L)

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 29 / 46

Page 83: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

LU Factorization Algorithm (3/3)

Step 6 Select lnn and unn satisfying lnnunn = ann −∑n−1

k=1 lnkukn

(Note: If lnnunn = 0, then A = LU but A is singular )

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 30 / 46

Page 84: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

LU Factorization Algorithm (3/3)

Step 6 Select lnn and unn satisfying lnnunn = ann −∑n−1

k=1 lnkukn

(Note: If lnnunn = 0, then A = LU but A is singular )

Step 7 OUTPUT (lij for j = 1, . . . , i and i = 1, . . . , n)OUTPUT (uij for j = i , . . . , n and i = 1, . . . , n)STOP

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 30 / 46

Page 85: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Using the LU Factorization to solve Ax = bOnce the matrix factorization is complete,

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 31 / 46

Page 86: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Using the LU Factorization to solve Ax = bOnce the matrix factorization is complete, the solution to a linearsystem of the form

Ax = LUx = b

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 31 / 46

Page 87: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Using the LU Factorization to solve Ax = bOnce the matrix factorization is complete, the solution to a linearsystem of the form

Ax = LUx = b

is found by first lettingy = Ux

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 31 / 46

Page 88: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Using the LU Factorization to solve Ax = bOnce the matrix factorization is complete, the solution to a linearsystem of the form

Ax = LUx = b

is found by first lettingy = Ux

and solvingLy = b

for y.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 31 / 46

Page 89: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Using the LU Factorization (Cont’d)

Since L is lower triangular, we have y1 =b1

l11

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 32 / 46

Page 90: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Using the LU Factorization (Cont’d)

Since L is lower triangular, we have y1 =b1

l11and, for each

i = 2, 3, . . . , n,

yi =1lii

bi −

i−1∑

j=1

lijyj

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 32 / 46

Page 91: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization

Using the LU Factorization (Cont’d)

Since L is lower triangular, we have y1 =b1

l11and, for each

i = 2, 3, . . . , n,

yi =1lii

bi −

i−1∑

j=1

lijyj

After y is found by this forward-substitution process, theupper-triangular system Ux = y is solved for x by backwardsubstitution using the equations

xn =yn

unnand xi =

1uii

yi −

n∑

j=i+1

uijxj

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 32 / 46

Page 92: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Outline

1 Computation Cost Rationale & Basic Solution Strategy

2 Constructing the Matrix Factorization

3 Example: LU Factorization of a 4 × 4 Matrix

4 The LU Factorization Algorithm

5 Permutation Matrices for Row Interchanges

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 33 / 46

Page 93: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Limitations of the LU Factorization AlgorithmWe assumed that Ax = b can be solved using Gaussianelimination without row interchanges.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 34 / 46

Page 94: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Limitations of the LU Factorization AlgorithmWe assumed that Ax = b can be solved using Gaussianelimination without row interchanges.

From a practical standpoint, this factorization is useful only whenrow interchanges are not required to control round-off error.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 34 / 46

Page 95: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Limitations of the LU Factorization AlgorithmWe assumed that Ax = b can be solved using Gaussianelimination without row interchanges.

From a practical standpoint, this factorization is useful only whenrow interchanges are not required to control round-off error.

We will now consider the modifications that must be made whenrow interchanges are required.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 34 / 46

Page 96: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

We begin with the introduction of a class of matrices that are used torearrange, or permute, rows of a given matrix.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 35 / 46

Page 97: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

We begin with the introduction of a class of matrices that are used torearrange, or permute, rows of a given matrix.

Permutation MatrixAn n × n permutation matrix P = [pij ] is a matrix obtained byrearranging the rows of In, the identity matrix. This gives a matrix withprecisely one nonzero entry in each row and in each column, and eachnonzero entry is a 1.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 35 / 46

Page 98: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

ExampleThe matrix

P =

1 0 00 0 10 1 0

is a 3× 3 permutation matrix.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 36 / 46

Page 99: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

ExampleThe matrix

P =

1 0 00 0 10 1 0

is a 3× 3 permutation matrix. For any 3× 3 matrix A, multiplying on theleft by P has the effect of interchanging the second and third rows of A:

PA =

1 0 00 0 10 1 0

a11 a12 a13

a21 a22 a23

a31 a32 a33

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 36 / 46

Page 100: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

ExampleThe matrix

P =

1 0 00 0 10 1 0

is a 3× 3 permutation matrix. For any 3× 3 matrix A, multiplying on theleft by P has the effect of interchanging the second and third rows of A:

PA =

1 0 00 0 10 1 0

a11 a12 a13

a21 a22 a23

a31 a32 a33

=

a11 a12 a13

a31 a32 a33

a21 a22 a23

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 36 / 46

Page 101: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

ExampleThe matrix

P =

1 0 00 0 10 1 0

is a 3× 3 permutation matrix. For any 3× 3 matrix A, multiplying on theleft by P has the effect of interchanging the second and third rows of A:

PA =

1 0 00 0 10 1 0

a11 a12 a13

a21 a22 a23

a31 a32 a33

=

a11 a12 a13

a31 a32 a33

a21 a22 a23

Similarly, multiplying A on the right by P interchanges the second andthird columns of A.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 36 / 46

Page 102: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Two useful properties of permutation matrices (1/2)Suppose k1, . . . , kn is a permutation of the integers 1, . . . , n

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 37 / 46

Page 103: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Two useful properties of permutation matrices (1/2)Suppose k1, . . . , kn is a permutation of the integers 1, . . . , n and thepermutation matrix P = (pij) is defined by

pij =

{

1, if j = ki

0, otherwise

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 37 / 46

Page 104: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Two useful properties of permutation matrices (2/2)Then

PA permutes the rows of A; that is,

PA =

ak11 ak12 · · · ak1n

ak21 ak22 · · · ak2n...

.... . .

...akn1 akn2 · · · aknn

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 38 / 46

Page 105: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Two useful properties of permutation matrices (2/2)Then

PA permutes the rows of A; that is,

PA =

ak11 ak12 · · · ak1n

ak21 ak22 · · · ak2n...

.... . .

...akn1 akn2 · · · aknn

P−1 exists and P−1 = P t .

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 38 / 46

Page 106: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Permutation Matrices & Gaussian EliminationEarlier, we saw that for any nonsingular matrix A, the linearsystem Ax = b can be solved by Gaussian elimination, with thepossibility of row interchanges.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 39 / 46

Page 107: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Permutation Matrices & Gaussian EliminationEarlier, we saw that for any nonsingular matrix A, the linearsystem Ax = b can be solved by Gaussian elimination, with thepossibility of row interchanges.

If we knew the row interchanges that were required to solve thesystem by Gaussian elimination, we could arrange the originalequations in an order that would ensure that no row interchangesare needed.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 39 / 46

Page 108: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Permutation Matrices & Gaussian EliminationEarlier, we saw that for any nonsingular matrix A, the linearsystem Ax = b can be solved by Gaussian elimination, with thepossibility of row interchanges.

If we knew the row interchanges that were required to solve thesystem by Gaussian elimination, we could arrange the originalequations in an order that would ensure that no row interchangesare needed.

Hence there is a rearrangement of the equations in the systemthat permits Gaussian elimination to proceed without rowinterchanges.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 39 / 46

Page 109: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Permutation Matrices & Gaussian Elimination (Cont’d)

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 40 / 46

Page 110: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Permutation Matrices & Gaussian Elimination (Cont’d)This implies that for any nonsingular matrix A, a permutationmatrix P exists for which the system

PAx = Pb

can be solved without row interchanges.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 40 / 46

Page 111: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Permutation Matrices & Gaussian Elimination (Cont’d)This implies that for any nonsingular matrix A, a permutationmatrix P exists for which the system

PAx = Pb

can be solved without row interchanges. As a consequence, thismatrix PA can be factored into PA = LU, where L is lowertriangular and U is upper triangular.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 40 / 46

Page 112: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Permutation Matrices & Gaussian Elimination (Cont’d)This implies that for any nonsingular matrix A, a permutationmatrix P exists for which the system

PAx = Pb

can be solved without row interchanges. As a consequence, thismatrix PA can be factored into PA = LU, where L is lowertriangular and U is upper triangular.

Because P−1 = P t , this produces the factorization

A = P−1LU = (P tL)U.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 40 / 46

Page 113: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Permutation Matrices & Gaussian Elimination (Cont’d)This implies that for any nonsingular matrix A, a permutationmatrix P exists for which the system

PAx = Pb

can be solved without row interchanges. As a consequence, thismatrix PA can be factored into PA = LU, where L is lowertriangular and U is upper triangular.

Because P−1 = P t , this produces the factorization

A = P−1LU = (P tL)U.

The matrix U is still upper triangular, but P tL is not lower triangularunless P = I.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 40 / 46

Page 114: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Example

Determine a factorization in the form A = (P tL)U for the matrix

A =

0 0 −1 11 1 −1 2

−1 −1 2 01 2 0 2

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 41 / 46

Page 115: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Example

Determine a factorization in the form A = (P tL)U for the matrix

A =

0 0 −1 11 1 −1 2

−1 −1 2 01 2 0 2

NoteThe matrix A cannot have an LU factorization because a11 = 0.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 41 / 46

Page 116: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Solution (1/4)However, using the row interchange (E1) ↔ (E2),

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 42 / 46

Page 117: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Solution (1/4)However, using the row interchange (E1) ↔ (E2), followed by(E3 + E1) → (E3)

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 42 / 46

Page 118: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Solution (1/4)However, using the row interchange (E1) ↔ (E2), followed by(E3 + E1) → (E3) and (E4 − E1) → (E4),

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 42 / 46

Page 119: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Solution (1/4)However, using the row interchange (E1) ↔ (E2), followed by(E3 + E1) → (E3) and (E4 − E1) → (E4), produces

1 1 −1 20 0 −1 10 0 1 20 1 1 0

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 42 / 46

Page 120: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Solution (1/4)However, using the row interchange (E1) ↔ (E2), followed by(E3 + E1) → (E3) and (E4 − E1) → (E4), produces

1 1 −1 20 0 −1 10 0 1 20 1 1 0

Then, the row interchange (E2) ↔ (E4),

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 42 / 46

Page 121: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Solution (1/4)However, using the row interchange (E1) ↔ (E2), followed by(E3 + E1) → (E3) and (E4 − E1) → (E4), produces

1 1 −1 20 0 −1 10 0 1 20 1 1 0

Then, the row interchange (E2) ↔ (E4), followed by (E4 + E3) → (E4),

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 42 / 46

Page 122: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Solution (1/4)However, using the row interchange (E1) ↔ (E2), followed by(E3 + E1) → (E3) and (E4 − E1) → (E4), produces

1 1 −1 20 0 −1 10 0 1 20 1 1 0

Then, the row interchange (E2) ↔ (E4), followed by (E4 + E3) → (E4),gives the matrix

U =

1 1 −1 20 1 1 00 0 1 20 0 0 3

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 42 / 46

Page 123: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Solution (2/4)The permutation matrix associated with the row interchanges

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 43 / 46

Page 124: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Solution (2/4)The permutation matrix associated with the row interchanges(E1) ↔ (E2) and (E2) ↔ (E4)

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 43 / 46

Page 125: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Solution (2/4)The permutation matrix associated with the row interchanges(E1) ↔ (E2) and (E2) ↔ (E4) is

P =

0 1 0 00 0 0 10 0 1 01 0 0 0

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 43 / 46

Page 126: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Solution (2/4)The permutation matrix associated with the row interchanges(E1) ↔ (E2) and (E2) ↔ (E4) is

P =

0 1 0 00 0 0 10 0 1 01 0 0 0

and

PA =

1 1 −1 21 2 0 2

−1 −1 2 00 0 −1 1

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 43 / 46

Page 127: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Solution (3/4)Gaussian elimination is performed on PA using the sameoperations as on A, except without the row interchanges.

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 44 / 46

Page 128: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Solution (3/4)Gaussian elimination is performed on PA using the sameoperations as on A, except without the row interchanges.

That is, (E2 − E1) → (E2), (E3 + E1) → (E3), followed by(E4 + E3) → (E4).

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 44 / 46

Page 129: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Solution (3/4)Gaussian elimination is performed on PA using the sameoperations as on A, except without the row interchanges.

That is, (E2 − E1) → (E2), (E3 + E1) → (E3), followed by(E4 + E3) → (E4).

The nonzero multipliers for PA are consequently,

m21 = 1, m31 = −1, and m43 = −1,

and the LU factorization of PA is

PA =

1 0 0 01 1 0 0

−1 0 1 00 0 −1 1

1 1 −1 20 1 1 00 0 1 20 0 0 3

= LU

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 44 / 46

Page 130: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Solution (4/4)

Multiplying by P−1 = P t produces the factorization

A = P−1(LU) = P t(LU) = (P tL)U

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 45 / 46

Page 131: Lu Factorization2

Rationale Constructing LU Example Algorithm Permutation Matrices

Matrix Factorization: Permutation Matrices

Solution (4/4)

Multiplying by P−1 = P t produces the factorization

A = P−1(LU) = P t(LU) = (P tL)U

=

0 0 −1 11 0 0 0

−1 0 1 01 1 0 0

1 1 −1 20 1 1 00 0 1 20 0 0 3

Numerical Analysis (Chapter 6) Matrix Factorization R L Burden & J D Faires 45 / 46

Page 132: Lu Factorization2

Questions?