ch3 (1).pdf

CHAPTER 3 (4 LECTURES)NUMERICAL SOLUTION OF SYSTEM OF LINEAR

EQUATIONS

1. Introduction

System of simultaneous linear equations are associated with many prob-lems in engineering and science, as well as with applications to the socialsciences and quantitative study of business and economic problems. Theseproblems occur in wide variety of disciplines, directly in real world problemsas well as in the solution process for other problems.The principal objective of this Chapter is to discuss the numerical aspectsof solving linear system of equations having the form

a11x1 + a12x2 + .........a1nxn = b1a21x1 + a22x2 + .........a2nxn = b2................................................an1x1 + an2x2 + .........annxn = bn.

(1.1)

This is a linear system of n equation in n unknowns x1, x2......xn. This systemcan simply be written in the matrix equation form

Ax=b

a11 a12 · · · a1na21 a22 · · · a2n...

.... . .

...an1 an2 · · · ann

×x1x2...xn

=

b1b2...bn

(1.2)

This equations has a unique solution x = A−1b, when the coefficient matrixA is non-singular. Unless otherwise stated, we shall assume that this is thecase under discussion. If A−1 is already available, then x = A−1b providesa good method of computing the solution x.If A−1 is not available, then in general A−1 should not be computed solelyfor the purpose of obtaining x. More efficient numerical procedures willbe developed in this chapter. We study broadly two categories Direct andIterative methods. We start with direct method to solve the linear system.

2. Gaussian Elimination

Direct methods, which are technique that give a solution in a fixed numberof steps, subject only to round-off errors, are considered in this chapter.Gaussian elimination is the principal tool in the direct solution of system

1

2 SOLUTION OF SYSTEM OF LINEAR EQUATIONS

(1.2). The method is named after Carl Friedrich Gauss (1777-1855). To solvelarger system of linear equation we use a method of introductory Algebra.For example, consider

x1 + 2x2 + x3 = 0

2x1 + 2x2 + 3x3 = 3

−x1 − 3x2 = 2.

Eliminating x1 from second and third equations, we obtain

x1 + 2x2 + x3 = 0

−2x2 + x3 = 3

−x2 + x3 = 2.

Now eliminate x2 from last equation with the help of second equation

x1 + 2x2 + x3 = 0

−2x2 + x3 = 31

2x3 = 2.

The last equation gives x3 = 1.Therefore −2x2 + 1 = 3, =⇒ x2 = −1.x1 + 2(−1) + 1 = 0, =⇒ x1 = 1. �

Formal structure of Gaussian elimination is the following.We consider a following 3× 3 system

a11x1 + a12x2 + a13x3 = b1 (R1)

a21x1 + a22x2 + a23x3 = b2 (R2)

a31x1 + a32x2 + a33x3 = b3 (R3).

Let a11 6= 0 and eliminate x1 from R2 and R3.

Define multipliers m21 =a21a11

and m31 =a31a11

.

We write each entry in R2 as R2 −m21R1 and in R3 as R3 −m31R1.We obtain a 2× 2 system

a(2)22 x2 + a

(2)23 x3 = b

(2)2 (R2)

a(2)32 x2 + a

(2)33 x3 = b

(2)3 (R3).

Here

a(2)ij = aij −mi1a1j , i, j = 2, 3

b(2)i = bi −mi1b1, i = 2, 3.

SOLUTION OF SYSTEM OF LINEAR EQUATIONS 3

Let a(2)22 6= 0 and eliminate x2 from R3.

Define multipliers m32 =a(2)33

a(2)22

. Subtract m32 times R2 from R3. This yields

a(3)33 x3 = b

(3)3 (R3).

Here

a(3)33 = a

(2)32 −m32a

(2)23 ,

b(3)3 = b

(2)3 −m32b

(2)2 .

Using back substitution, we can find the values

x3 =b(3)3

a(3)33

x2 =b(2)3 − a

(2)23 x3

a(2)22

x1 =b1 − a12x2 − a13x3

a11.

Note that every step of the elimination procedure can be obtained throughelementary row operations on the Augmented matrix (A | b).

Algorithm [Gauss Elimination]

(1) Start(2) Declare the variables and read the order of the matrix n.(3) Input the coefficients of the linear equation with right side as:

Do for i = 1 to nDo for j = 1 to n+ 1Read a[i][j] End for jEnd for i

(4) Do for k = 1 to n− 1Do for i = k + 1 to nDo for j = k + 1 to n+ 1a[i][j] = a[i][j]− a[i][k]/a[k][k] ∗ a[k][j] End for jEnd for iEnd for k

(5) Compute x[n] = a[n][n+ 1]/a[n][n](6) Do for i = n− 1 to 1

sum = 0Do for j = i+ 1 to nsum = sum +a[i][j] ∗ x[j] End for jx[i] = 1/a[i][i] ∗ (a[i][n+ 1]− sum)End for i

(7) Display the result x[i](8) Stop


Partial Pivoting: In the elimination process, it is assumed that pivot ele-ment aii 6= 0, i = 1, 2, . . . , n. If at any stage of elimination, one of the pivotbecomes small (or zero) then we bring other element as pivot by interchang-ing rows.

Remark 2.1. Unique Solution, No Solution, or Infinite Solutions.

Here are some tips that will allow us to determine what type of solutionswe have based on either the reduced echelon form.1. If we have a leading one in every column, then we will have a uniquesolution.2. If we have a row of zeros equal to a non-zero number in right side, thenthe system has no solution.3. If we don’t have a leading one in every column in a homogeneous system,i.e. a system where all the equations equal zero, or a row of zeros, thensystem have infinite solutions.

Example 1. Solve the system of equations. This system has solution x1 =2.6, x2 = −3.8, x3 = −5.0.

6x1 + 2x2 + 2x3 = −2

2x1 +2

3x2 +

1

3x3 = 1

x1 + 2x2 − x3 = 0.

Sol. Let us use a floating-point representation with 4−digits and all opera-tions will be rounded. Augmented matrix is given by6.000 2.000 2.000 −2.000

2.000 1.667 0.3333 1.0001.000 2.000 −1.000 0.0

Multipliers are m21 =

2

6= 0.3333 and m31 =

1

6= 0.1667.

a(2)21 = a21 −m21a11, a

(2)22 = a22 −m21a12 etc.6.000 2.000 2.000 −2.000

0.0 0.0001000 −0.3333 1.6670.0 1.667 −1.333 0.3334

Multiplier is m32 =

1.667

0.0001= 166706.000 2.000 2.000 −2.000

0.0 0.0001000 −0.3333 1.6670.0 0.0 5555 −27790

Using back substitution, we obtain

x3 = −5.003

x2 = 0.0


x1 = 1.335.

We observe that computed solution is not compatible with the exact solu-tion.The difficulty is in a22. This coefficient is very small (almost zero). Thismeans that the coefficient in this position had essentially infinite relative er-ror and this was carried through into computation involving this coefficient.To avoid this, we interchange second and third rows and then continue theelimination.In this case (after interchanging) multipliers is m32 = 0.00005999.6.000 2.000 2.000 −2.000

0.0 1.667 −1.337 0.33340.0 0.0 −0.3332 1.667

Using back substitution, we obtain

x3 = −5.003

x2 = −3.801

x1 = 2.602.

We see that after partial pivoting, we get the desired solution.Complete Pivoting: In the first stage of elimination, we search the largestelement in magnitude from the entire matrix and bring it at the position offirst pivot. We repeat the same process at every step of elimination. Thisprocess require interchange of both rows and columns.Scaled Partial Pivoting: In this approach, the algorithm selects as thepivot element the entry that has the largest relative entries in its rows.At the beginning, a scale factor must be computed for each equation in thesystem. We define

si = max1≤j≤n

|aij | (1 ≤ i ≤ n)

These numbers are recored in the scaled vector s = [s1, s2, · · · , sn]. Notethat the scale vector does not change throughout the procedure. In startingthe forward elimination process, we do not arbitrarily use the first equation

as the pivot equation. Instead, we use the equation for which the ratioai,1si

is greatest. We repeat the process by taking same scaling factors.

Example 2. Solve the system

3x1 − 13x2 + 9x3 + 3x4 = −19

−6x1 + 4x2 + x3 − 18x4 = −34

6x1 − 2x2 + 2x3 + 4x4 = 16

12x1 − 8x2 + 6x3 + 10x4 = 26

by hand using scaled partial pivoting. Justify all row interchanges and writeout the transformed matrix after you finish working on each column.


Sol. The augmented matrix is3 −13 9 3 −19−6 4 1 −18 −346 −2 2 4 1612 −8 6 10 26.

and the scale factors are s1 = 13, s2 = 18, s3 = 6,& s4 = 12. We need to pickthe largest (3/13, 6/18, 6/6, 12/12), which is the third entry, and interchangerow 1 and row 3 and interchange s1 and s3 to get

6 −2 2 4 16−6 4 1 −18 −343 −13 9 3 −1912 −8 6 10 26.

with s1 = 6, s2 = 18, s3 = 13, s4 = 12. Performing R2 − (−6/6)R1 → R2,R3 − (3/6)R1 → R3, and R4 − (12/6)R1 → R4, we obtain

6 −2 2 4 160 2 3 −14 −180 −12 8 1 −270 −4 2 2 −6

Comparing (a22/s2 = 2/18, a32/s3 = 12/13, a42/s4 = 4/12), the largest isthe third entry so we need to interchange row 2 and row 3 and interchanges2 and s3 to get

6 −2 2 4 160 −12 8 1 −270 2 3 −14 −180 −4 2 2 −6

with s1 = 6, s2 = 13, s3 = 18, s4 = 12. Performing R3− (2/12)R2 → R3 andR4 − (−4/12)R2 → R4, we get

6 −2 2 4 160 −12 8 1 −270 0 13/3 −83/6 −45/20 0 −2/3 5/3 3

Comparing (a33/s3 = (13/3)/18, a43/s4 = (2/3)/12), the largest is the firstentry so we do not interchange rows. Performing R4− (−2/13)R3 → R4, weget the final reduced matrix

6 −2 2 4 160 −12 8 1 −270 0 13/3 −83/6 −45/20 0 0 −6/13 −6/13

Backward substitution gives x1 = 3, x2 = 1, x3 = −2, x4 = 1.


Example 3. Solve this system of linear equations:

0.0001x+ y = 1

x+ y = 2

using no pivoting, partial pivoting, and scaled partial pivoting. Carry atmost five significant digits of precision (rounding) to see how finite precisioncomputations and roundoff errors can affect the calculations.

Sol. By direct substitution, it is easy to verify that the true solution isx = 1.0001 and y = 0.99990 to five significant digits.For no pivoting, the first equation in the original system is the pivot equa-tion, and the multiplier is 1/0.0001 = 10000. The new system of equationsis

0.0001x+ y = 1

9999y = 9998

We obtain y = 9998/9999 ≈ 0.99990 and x = 1. Notice that we have lostthe last significant digit in the correct value of x.We repeat the solution process using partial pivoting for the original system.We see that the second entry is larger, so the second equation is used as thepivot equation. We can interchange the two equations, obtaining

x+ y = 2

0.0001x+ y = 1

which gives y = 0.99980/0.99990 ≈ 0.99990 and x = 2− y = 2− 0.99990 =1.0001.Both computed values of x and y are correct to five significant digits.We repeat the solution process using scaled partial pivoting for the originalsystem. Since the scaling constants are s = (1, 1) and the ratios for deter-mining the pivot equation are (0.0001/1, 1/1), the second equation is nowthe pivot equation. We do not actually interchange the equations and usethe second equation as the first pivot equation. The rest of the calculationsare as above for partial pivoting. The computed values of x and y are correctto five significant digits.Operations count for Gauss elimination We consider the number offloating point operations (“flops”) required to solve the system Ax = b.Gaussian Elimination first uses row operations to transform the probleminto an equivalent problem of the form Ux = b, where U is upper triangular.Then back substitution is used to solve for x. First we look at how manyfloating point operations are required to reduce into upper triangular matrix.First a multiplier is computed for each row. Then in each row the algorithmperforms n multiplies and n adds. This gives a total of (n − 1) + (n − 1)nmultiplies (counting in the computing of the multiplier in each of the (n−1)rows) and (n− 1)n adds.In total this is 2n2 − n− 1 floating point operations to do a single pivot on


the n by n system.Then this has to be done recursively on the lower right subsystem, whichis an (n − 1) by (n − 1) system. This requires 2(n − 1)2 − (n − 1) − 1operations. Then this has to be done on the next subsystem, requiring2(n− 2)2 − (n− 2)− 1 operations, and so on.In total, then, we use In total floating point operations, with

In =n∑

k=1

(2k2 − k − 1) =n(n+ 1)(4n− 1)

6− n ≈ 2

3n3.

Counts for back substitution: To find xn we just requires one division. Thento solve for xn−1 we requires 3 flops. Similarly, solving for xn−2 requires5 flops. Thus in total back substitution requires Bn total floating pointoperations with

Bn =n∑

k=1

(2k − 1) = n(n− 1)− n = n(n− 2) ≈ n2.

The LU Factorization: When we use matrix multiplication, anothermeaning can be given to the Gauss elimination. The matrix A can befactored into the product of the two triangular matrices.Let AX = b is the system to be solved, A is n × n coefficient matrix. Thelinear system can be reduced to the upper triangular system UX = g with

U =

u11 u12 · · · u1n0 u22 · · · u2n...

. . ....

0 0 · · · unn

Here uij = aij . Introduce an auxiliary lower triangular matrix L based onthe multipliers mij as following

L =

1 0 0 · · · 0m21 1 0 · · · 0m31 m32 1 · · · 0

.... . .

...mn,1 0 · · · mn,n−1 1

Theorem 2.1. Let A be a non-singular matrix and let L and U be definedas above. If U is produced without pivoting then

LU = A.

This is called LU factorization of A.


Example 4. Solve the system

4x1 + 3x2 + 2x3 + x4 = 1

3x1 + 4x2 + 3x3 + 2x4 = 1

2x1 + 3x2 + 4x3 + 3x4 = −1

x1 + 2x2 + 3x3 + 4x4 = −1.

Also write the LU decomposition and verify that LU = A.

Sol. We write augmented matrix and solve the system4 3 2 1 13 4 3 2 12 3 4 3 −11 2 3 4 −1


3

4, m31 =

1

2, and m41 =

1

4.

Replace R2 with R2−m21R1, R3 with R3−m31R1 and R4 with R4−m41R1.4 3 2 1 10 7/4 3/2 5/4 1/40 3/2 3 5/2 −3/20 5/4 5/2 15/4 −5/4


6

7and m42 =

5

7.

Replace R3 with R3 −m32R2 and R4 with R4 −m42R2, we obtain4 3 2 1 10 7/4 3/2 5/4 1/40 0 12/7 10/7 −12/70 0 10/7 20/7 −10/7

Multiplier is m43 =

5

6and we replace R4 with R4 −m43R3.4 3 2 1 10 7/4 3/2 5/4 1/40 0 12/7 10/7 −12/70 0 0 5/3 0

Using back substitution successively for x4, x3, x2, x1, we obtain x4 = 0,x3 = −1, x2 = 1, x1 = 0.Here

U =

4 3 2 10 7/4 3/2 5/40 0 12/7 10/70 0 0 5/3


L =

1 0 0 0

3/4 1 0 01/2 6/7 1 01/4 5/7 5/6 1

It can be verified that LU = A.

3. Iterative Method

The linear system Ax = b may have a large order. For such systems Gausselimination is often too expensive in either computation time or computermemory requirements or both.In an iterative method, a sequence of progressively iterates is produced toapproximate the solution.Jacobi and Gauss-Seidel Method: We start with an example and let usconsider a system of equations

9x1 + x2 + x3 = 10

2x1 + 10x2 + 3x3 = 19

3x1 + 4x2 + 11x3 = 0.

One class of iterative method for solving this system as follows.We write

x1 =1

9(10− x2 − x3)

x2 =1

10(19− 2x1 − 3x3)

x3 =1

11(0− 3x1 − 4x2).

Let x(0) = [x(0)1 x

(0)2 x

(0)3 ] be an initial approximation of solution x. Then

define an iteration of sequence

x(k+1)1 =

1

9(10− x(k)2 − x

(k)3 )

x(k+1)2 =

1

10(19− 2x

(k)1 − 3x

(k)3 )

x(k+1)3 =

1

11(0− 3x

(k)1 − 4x

(k)2 ), k = 0, 1, 2, . . . .

This is called Jacobi or method of simultaneous replacements. The methodis named after German mathematician Carl Gustav Jacob Jacobi.We start with [0 0 0] and obtain

x(1)1 = 1.1111, x

(1)2 = 1.900, x

(1)3 = 0.0.

x(2)1 = 0.9000, x

(2)2 = 1.6778, x

(2)3 = −0.9939

etc.An another approach to solve the same system will be following.

x(k+1)1 =

1

9(10− x(k)2 − x

(k)3 )


x(k+1)2 =

1

10(19− 2x

(k+1)1 − 3x

(k)3 )

x(k+1)3 =

1

11(0− 3x

(k+1)1 − 4x

(k+1)2 ), k = 0, 1, 2, . . . .

This method is called Gauss-Seidel or method of successive replacements.It is named after the German mathematicians Carl Friedrich Gauss andPhilipp Ludwig von Seidel. Starting with [0 0 0], we obtain

x(1)1 = 1.1111, x

(1)2 = 1.6778, x

(1)3 = −0.9131.

x(2)1 = 1.0262, x

(2)2 = 1.9687, x

(2)3 = −0.9588.

General approach: We rewrite the system Ax = b as

a11x(k+1)1 = −a12x(k)2 − · · ·+ b1

a21x(k+1)1 + a22x

(k+1)2 = −a23x(k)3 − · · ·+ b2

..................................

an1x(k+1)1 + an2x

(k+1)2 + · · ·+ annx

(k+1)n = bn

or (D + L)x(k+1) = −Ux(k) + b

where D, L and U are diagonal, strictly lower triangle and upper trianglematrices, respectively.

x(k+1) = −(D + L)−1Ux(k) + (D + L)−1b

x(k+1) = T x(k) +B, k = 0, 1, 2, · · ·Here T = −(D + L)−1U and this matrix is called iteration matrix.

Algorithm[Gauss-Seidel]

(1) Input matrix A = [aij ], b,XO = x(0), tolerance TOL, maximumnumber of iterations

(2) Set k = 1(3) while (k ≤ N) do step 4-7(4) For i = 1, 2, · · · , n

xi =1

aii

− i−1∑j=1

(aijxj)−n∑

j=i+1

(aijXOj) + bi)

(5) If ||x−XO|| < TOL, then OUTPUT (x1, x2, · · · , xn)

STOP(6) k = k + 1(7) For i = 1, 2, · · · , n

Set XOi = xi(8) OUTPUT (x1, x2, · · · , xn)

STOP.


3.1. Convergence analysis of iterative methods.

Definition 3.1 (Eigenvalue and eigenvector). Let A be a square matrix thennumber λ is called an eigenvalue of A if there exists a nonzero vector x suchthat Ax = λx. Here x is called the corresponding eigenvector.

Definition 3.2 (Characteristic polynomial). Characteristic polynomial isdefined as

P (λ) = det(λI −A).

λ is an eigenvalue of matrix A if and only if λ is a root of the characteristicpolynomial, i.e., P (λ) = 0.

Definition 3.3 (Spectrum and spectral radius). The set of all eigenvaluesof matrix A is called the spectrum of A and is denoted by λ(A).The spectrum radius of A is defined as

ρ(A) = max{ |λ|, λ ∈ λ(A) }.

Theorem 3.4 (Necessary and sufficient condition). A necessary and suffi-cient condition for the convergence of an iterative method is that the eigen-value of iteration matrix T satisfy the inequality

ρ(T ) < 1.

Proof. Letρ(T ) < 1.

The sequence of vector x(k) by iterative method (Gauss-Seidel) are given by

x(1) = Tx(0) +B.

x(2) = Tx(1) +B = T (Tx(0) +B) +B = T 2x(0) + (T + I)B.

........................

x(k) = T kx(0) + (T k−1 + T k−2 + ...+ T + I)B

Since ρ(T ) < 1, this implies

limk→∞

T kx(0) = 0

Thereforelimk→∞

x(k) = (I − T )−1B as k →∞

Therefore, x(k) converges to unique solution x = Tx+B.Conversely, assume that the sequence x(k) converges to x. Now

x− x(k) = Tx+B − Tx(k−1) −B = T (x− x(k−1))

= T 2(x− x(k−2))= T k(x− x(0).

Let z = x− x(0)) then

limk→∞

T kz = limk→∞

(x− x(k)) = x− limk→∞

x(k) = x− x = 0.

=⇒ ρ(T ) < 1.


Theorem 3.5. If A is strictly diagonally dominant in Ax = b, then Gauss-Seidel iteration always converges for any initial starting vector.

Proof. We assume that A is strictly diagonally dominant, hence aii 6= 0and

|aii| >n∑

j=1,j 6=i

|aij |, i = 1, 2, · · · , n

Gauss-Seidel iterations are given by

x(k) = −(D + L)−1Ux(k−1) + (D + L)−1

x(k) = Tx(k−1) +B.

Method is convergent iff ρ(T ) < 1, where T = −(D + L)−1U .Let λ be an eigenvalue of matrix A and x be an eigenvector then

Tx = λx

−(D + L)−1Ux = λx

−Ux = λ(D + L)x

−∑

j=i+1

naij = λ[i∑

j=1

aijxj ], i = 1, 2, . . . , n

−∑

j=i+1

naij = λaiixi + λi−1∑j=1

aijxj

λaiixi = −λi−1∑j=1

aijxj − λn∑

j=i+1

aijxj

|λaiixi| ≤ |λ|i−1∑j=1

|aij | |xj |+ |λ|n∑

j=i+1

|aij | |xj |

Since x is an eigenvector, x 6= 0, so we can take norm ||x||∞ = 1.Hence

|λ|

|aii| − i−1∑j=1

|aij |

≤ n∑j=i+1

|aij |

=⇒ |λ| ≤∑n

j=i+1 |aij ||aii| −

∑i−1j=1 |aij |

≤∑n

j=i+1 |aij |∑nj=i+1 |aij |

= 1

which implies spectral radius ρ(T ) < 1.This implies Gauss-Seidel is convergent.


Example 5. Given the matrix

A =

1 2 −21 1 12 2 1

Decide whether Gauss-Seidel converge to the solution of Ax = b.

Sol. The iteration matrix of the Gauss-Seidel method is

T = −(D + L)−1U

= −

1 0 01 1 02 2 1

−1 0 2 −20 0 10 0 0

= −

1 0 0−1 1 00 −2 1

0 2 −20 0 10 0 0

= −

0 2 −20 −2 30 0 −2

=

0 −2 20 2 −30 0 −2

The eigenvalues of iteration matrix T are λ = 0, 2, 2 and therefore spectralradius > 1. The iteration diverges.

4. Power method for approximating eigenvalues

The eigenvalues of an n × n of matrix A are obtained by solving itscharacteristic equation

det(a− λI) = 0

λn + cn−1λn−1cn−2λ

n−2 + · · ·+ c0 = 0.

For large values of n, the polynomial equations like this one are difficult,time-consuming to solve and sensitive to rounding errors. In this section welook at an alternative method for approximating eigenvalues. The methodcan be used only to find the eigenvalue of A that is largest in absolute value.We call this eigenvalue the dominant eigenvalue of A.

Definition 4.1 (Dominant Eigenvalue and Dominant Eigenvector). Let λ1,λ2, · · · , and λn be the eigenvalues of an n × n matrix A. λ1 is called thedominant eigenvalue of A if

|λ1| > |λi|, i = 2, 3, . . . , n.

The eigenvectors corresponding to λ1 are called dominant eigenvectors of A.


4.1. The Power Method. The power method for approximating eigen-values is iterative. First we assume that the matrix A has a dominanteigenvalue with corresponding dominant eigenvectors. Then we choose aninitial approximation x0 of one of the dominant eigenvectors of A. Thisinitial approximation must be a nonzero vector in R. Finally we form thesequence given by

x1 = Ax0

x2 = Ax1 = A2x0

..............

xk = Axk−1 = A(Ak−1x0) = Akx0.

For large powers of k, and by properly scaling this sequence, we will seethat we obtain a good approximation of the dominant eigenvector of A.This procedure is illustrated in the following example.

Example 6. Complete six iterations of the power method to approximate adominant eigenvector of

A =

[2 −121 −5

].

Sol. We begin with an initial non-zero approximation of

x0 =

[11

]We then obtain the following approximations by power method

y1 = Ax0 =

[2 −121 −5

] [11

]=

[−10−4

]= −10

[1.000.40

]= −10x1

y2 = Ax1 =

[2 −121 −5

] [1.000.40

]=

[−2.8−1

]= −2.8

[1.0000.357

]= −2.8x2

y3 = Ax2 =

[2 −121 −5

] [1.0000.357

]=

[−2.284−0.785

]= −2.284

[1.0000.3436

]After several iterations, we note that the approximations appears to be ap-proaching the dominant eigenvalue λ = −2.

Theorem 4.2. If A is an n × n diagonalizable matrix with a dominanteigenvalue, then there exists a nonzero vector x0 such that the sequence ofvectors given by

Ax0, A2x0, A

3x0, . . . , Akx0, . . .

approaches a multiple of the dominant eigenvector of A.

Proof. Let A is diagonalizable, which implies that it has n linearlyindependent eigenvectors x1, x2, . . . , xn with corresponding eigenvalues ofλ1, λ2, · · · , λn.We assume that these eigenvalues are ordered so that λ1 is the dominanteigenvalue (with a corresponding eigenvector of x1).


Because the n eigenvectors x1, x2, . . . , xn are linearly independent, they mustform a basis for Rn.For the initial approximation x0, we choose a nonzero vector such that thelinear combination

x0 = c1x1 + c2x2 + · · ·+ cnxn

has nonzero leading coefficients. (If c1 = 0, the power method may notconverge, and a different x0 must be used as the initial approximation.Now, operating both sides of this equation by A produces

Ax0 = Ac1x1 +Ac2x2 + · · ·+Acnxn

Ax0 = c1(Ax1) + c2(Ax2) + · · ·+ cn(Axn)

Ax0 = c1(λ1x1) + c2(λ2x2) + · · ·+ cn(λn2xn)

As λi are eigenvalues, so Axi = λixi.Repeated multiplication of both sides of this equation by A produces

Akx0 = c1(λk1x1) + c2(λ

k2x2) + · · ·+ cn(λknxn)

which implies that

Akx0 = λk1

[c1x1 + c2

(λ2λ1

)k

x2 + · · ·+ cn

(λnλ1

)k

xn

]Now, from our original assumption that λ1 is larger in absolute value thanthe other eigenvalues it follows that each of the fractions

λ2λ1,λ3λ1, · · · , λn

λ1< 1.

Therefore each of the factors(λ2λ1

)k

,

(λ3λ1

)k

, · · · ,(λnλ1

)k

must approach 0 as k approaches infinity. This implies that the approxima-tion

Akx0 ≈ λk1c1x1, c1 6= 0

improves as k increases. Since x1 is a dominant eigenvector, it follows thatany scalar multiple of x1 is also a dominant eigenvector. Thus we haveshown that Akx0 approaches a multiple of the dominant eigenvector of A.

Algorithm

(1) Start(2) Define matrix A and initial guess x(3) Calculate y = Ax(4) Find the largest element in magnitude of matrix y and assign it to

K.(5) Calculate fresh value x = (1/K) ∗ y(6) If [K(n)−K(n− 1)] > error, goto step 3.(7) Stop


Example 7. Calculate seven iterations of the power method with scaling toapproximate a dominant eigenvector of the matrix 1 2 0

−2 1 21 3 1

Sol. Using x0 = [1, 1, 1]T as initial approximation, we obtain

y1 = Ax0 =

1 2 0−2 1 21 3 1

111

=

315

and by scaling we obtain the approximation

x1 = 1/5

315

=

0.600.201.00

Similarly we get

y2 = Ax1 =

1.001.002.20

= 2.20

0.450.451.00

= 2.20x2

y3 = Ax2 =

1.351.552.8

= 2.8

0.480.551.00

= 2.8x3

y4 = Ax3 = 3.1

0.510.511.00

etc.After several iterations, we observe that dominant eigenvector is

x =

0.500.501.00

Scaling factors are approaching to dominant eigenvalue λ = 3.

Remark 4.1. The power method is useful to compute the eigenvalue but itgives only dominant eigenvalue. To find other eigenvalue we use propertiesof matrix such as sum of all eigenvalue is equal to the trace of matrix. Alsoif λ is an eigenvalue of A then λ−1 is the eigenvalue of A−1. Hence thesmallest eigenvalue of A is the dominant eigenvalue of A−1.

Remark 4.2. We consider A − σI then its eigenvalues are (λ1 − σ, λ2 −σ, . . . ). Now the eigenvalues of (A− σI)−1 are (

1

λ1 − σ,

1

λ2 − σ, . . . ).

The eigenvalues of the original matrix A that is the closest to σ correspondsto the eigenvalue of largest magnitude of the shifted and inverted of matrix(A− σI)−1.


To find the eigenvalue closest to σ, we apply the Power method to obtainthe eigenvalue µ of (A − σI)−1. Then we recover the eigenvalue λ of theoriginal problem by λ = 1/µ + σ. This method is shifted and inverted. Wesolve y = (A−σI)−1x which implies (A−σI)y = x. We need not to computethe inverse of the matrix.

Example 8. Find the eigenvalue of matrix nearest to 3 2 −1 0−1 2 −10 −1 2

using power method.

Sol. The eigenvalue of matrix A which is nearest to 3 is the smallest eigen-value in magnitude of A−3I. Hence it is the largest eigenvalue of (A−3I)−1

in magnitude. Now

A− 3I =

−1 −1 0−1 −1 −10 −1 −1

B = (A− 3I)−1 =

0 −1 1−1 1 −11 −1 0

.Starting with

x0 =

111

we obtain

y1 = Bx0 =

0 −1 1−1 1 −11 −1 0

111

=

0−10

= 1.x1

y2 = Bx1 =

1−11

= 1.x2

y3 = Bx2 =

2−32

= 3

0.6667−1

0.6667

= 3x3

y4 = Bx3 =

1.6667−2.33341.6667

= 2.3334

0.7143−1

0.7143

= 2.3334x4.

After six iterations, we obtain the dominant eigenvalue of matrix B andwhich is 2.4 and the dominant eigenvector is0.7143

−10.7143

.


Now the eigenvalue of matrix A is 3± 12.4 = 3±0.42 = 3.42, 2.58. Since 2.58

does not satisfy |A − 2.58I| = 0, therefore the correct eigenvalue of matrixA nearest to 3 is 3.42.

Although the power method worked well in these examples, we must saysomething about cases in which the power method may fail. There are ba-sically three such cases :1. Using the power method when A is not diagonalizable. Recall that Ahas n linearly Independent eigenvector if and only if A is diagonalizable. Ofcourse, it is not easy to tell by just looking at A whether is is diagonalizable.2. Using the power method when A does not have a dominant eigenvalue orwhen the dominant eigenvalue is such that λ1 = λ2.3. If the entries of A contains significant error. Powers Ak will have signifi-cant roundoff error in their entires.

Exercises

(1) Using the four-decimal-place computer solve the following system ofequation without and with pivoting

0.729x1 + 0.81x2 + 0.9x3 = 0.6867

x1 + x2 + x3 = 0.8338

1.331x1 + 1.21x2 + 1.1x3 = 1.000

This system has exact solution, rounded to four places x1 = 0.2245,x2 = 0.2814, x3 = 0.3279.

(2) Solve the following system of equations by Gaussian elimination withpartial and scaled partial pivoting

x1 + 2x2 + x3 = 3

3x1 + 4x2 = 3

2x1 + 10x2 + 4x3 = 10.

(3) Consider the linear system

x1 + 4x2 = 1

4x1 + x2 = 0.

The true solution is x1 = −1/5 and x2 = 4/15. Apply the Jacobi

and Gauss-Seidel methods with x(0) = [0, 0]T to the system andfind out which methods diverges rapidly. Next, interchange the twoequations to write the system as

4x1 + x2 = 0

x1 + 4x2 = 1

and apply both methods with x(0) = [0, 0]T .

Iterate until ||x− x(k)|| ≤ 10−5. Which method converges faster?


(4) Solve the system of equations by Jacobi and Gauss-Seidel method

−8x1 + x2 + 2x3 = 1

x1 − 5x2 + x3 = 16

x1 + x2 − 4x3 = 7.

(5) Solve this system of equations by Gauss-Seidel, starting with theinitial vector [0,0,0]:

4.63x1 − 1.21x2 + 3.22x3 = 2.22

−3.07x1 + 5.48x2 + 2.11x3 = −3.17

1.26x1 + 3.11x2 + 4.57x3 = 5.11.

(6) Show that Gauss-Seidel method does not converge for the followingsystem of equations

2x1 + 3x2 + x3 = −1

3x1 + 2x2 + 2x3 = 1

x1 + 2x2 + 2x3 = 1.

(7) Consider the iteration

x(k+1) = b+ α

[2 11 2

]x(k), k ≥ 0

where α is a real constant. For some values of α, the iterationmethod converges for any choice of initial guess x(0), and for someother values of α, the method diverges. Find the values of α forwhich the method converges.

(8) Determine the largest eigenvalue and the corresponding eigenvectorof the matrix

A =

4 1 01 20 10 1 4

correct to three decimals using the power method.

(9) Find the smallest eigenvalue in magnitude of the matrix 2 −1 0−1 2 −10 −1 2

using four iterations of the inverse power method.

Bibliography

[Gerald] Curtis F. Gerald and Patrick O. Wheatley “Applied NumericalAnalysis,” 7th edition, Pearson, 2003.

[Atkinson] K. Atkinson and W. Han. “Elementary Numerical Analysis,”3rd edition, John Willey and Sons, 2004.

ch3 (1).pdf

Documents