linear systems

Linear SystemsLinear Systems

COS 323COS 323


3333232131

2323222121

1313212111

bxaxaxa

bxaxaxa

bxaxaxa

3333232131

2323222121

1313212111

bxaxaxa

bxaxaxa

bxaxaxa

3

2

1

3

2

1

333231

232221

131211

b

b

b

x

x

x

aaa

aaa

aaa

3

2

1

3

2

1

333231

232221

131211

b

b

b

x

x

x

aaa

aaa

aaa


• Solve Ax=b, where A is an Solve Ax=b, where A is an nnnn matrix and matrix andb is an b is an nn1 column vector1 column vector

• Can also talk about non-square systems Can also talk about non-square systems wherewhereA is A is mmnn, b is , b is mm1, and x is 1, and x is nn11– OverdeterminedOverdetermined if if mm>>nn::

“more equations than unknowns”“more equations than unknowns”

– UnderdeterminedUnderdetermined if if nn>>mm::“more unknowns than equations”“more unknowns than equations”Can look for best solution using least squaresCan look for best solution using least squares

Singular SystemsSingular Systems

• A is singular if some row isA is singular if some row islinear combination of other rowslinear combination of other rows

• Singular systems can be Singular systems can be underdetermined:underdetermined:

or inconsistent:or inconsistent:

1064

532

21

21

xx

xx

1064

532

21

21

xx

xx

1164

532

21

21

xx

xx

1164

532

21

21

xx

xx

Inverting a MatrixInverting a Matrix

• Usually not a good idea to compute x=AUsually not a good idea to compute x=A-1-1bb– InefficientInefficient

– Prone to roundoff errorProne to roundoff error

• In fact, compute inverse using linear solverIn fact, compute inverse using linear solver– Solve AxSolve Axii=b=bii where b where bii are columns of identity, are columns of identity,

xxii are columns of inverse are columns of inverse

– Many solvers can solve several R.H.S. at onceMany solvers can solve several R.H.S. at once

Gauss-Jordan EliminationGauss-Jordan Elimination

• Fundamental operations:Fundamental operations:1.1. Replace one equation with linear combinationReplace one equation with linear combination

of other equationsof other equations

2.2. Interchange two equationsInterchange two equations

3.3. Re-label two variablesRe-label two variables

• Combine to reduce to trivial systemCombine to reduce to trivial system

• Simplest variant only uses #1 operations,Simplest variant only uses #1 operations,but get better stability by addingbut get better stability by adding#2 (partial pivoting) or #2 and #3 (full #2 (partial pivoting) or #2 and #3 (full pivoting)pivoting)


• Solve:Solve:

• Only care about numbers – form Only care about numbers – form “tableau” or “augmented matrix”:“tableau” or “augmented matrix”:

1354

732

21

21

xx

xx

1354

732

21

21

xx

xx

13

7

54

32

13

7

54

32


• Given:Given:

• Goal: reduce this to trivial systemGoal: reduce this to trivial system

and read off answer from right columnand read off answer from right column

13

7

54

32

13

7

54

32

?

?

10

01

?

?

10

01


• Basic operation 1: replace any row byBasic operation 1: replace any row bylinear combination with any other rowlinear combination with any other row

• Here, replace row1 with Here, replace row1 with 11//22 * row1 + 0 * * row1 + 0 *

row2row2

13

7

54

32

13

7

54

32

1354

1 27

23

1354

1 27

23


• Replace row2 with row2 – 4 * row1Replace row2 with row2 – 4 * row1

• Negate row2Negate row2

1354

1 27

23

1354

1 27

23

110

1 27

23

110

1 27

23

110

1 27

23

110

1 27

23


• Replace row1 with row1 – Replace row1 with row1 – 33//22 * row2 * row2

• Read off solution: xRead off solution: x1 1 = 2, x= 2, x2 2 = 1= 1

110

1 27

23

110

1 27

23

1

2

10

01

1

2

10

01


• For each row i:For each row i:– Multiply row i by 1/aMultiply row i by 1/aiiii

– For each other row j:For each other row j:

• Add –aAdd –ajiji times row i to row j times row i to row j

• At the end, left part of matrix is identity,At the end, left part of matrix is identity,answer in right partanswer in right part

• Can solve any number of R.H.S. Can solve any number of R.H.S. simultaneouslysimultaneously

PivotingPivoting

• Consider this system:Consider this system:

• Immediately run into problem:Immediately run into problem:algorithm wants us to divide by zero!algorithm wants us to divide by zero!

• More subtle version:More subtle version:

8

2

32

10

8

2

32

10

8

2

32

1001.0

8

2

32

1001.0

PivotingPivoting

• Conclusion: small diagonal elements Conclusion: small diagonal elements badbad

• Remedy: swap in larger element from Remedy: swap in larger element from somewhere elsesomewhere else

Partial PivotingPartial Pivoting

• Swap rows 1 and 2:Swap rows 1 and 2:

• Now continue:Now continue:

8

2

32

10

8

2

32

10

2

8

10

32

2

8

10

32

2

1

10

01

2

4

10

1 23

2

1

10

01

2

4

10

1 23

Full PivotingFull Pivoting

• Swap largest element onto diagonal by Swap largest element onto diagonal by swapping rows 1 and 2 and columns 1 swapping rows 1 and 2 and columns 1 and 2:and 2:

• Critical: when swapping columns, must Critical: when swapping columns, must remember to swap results!remember to swap results!

8

2

32

10

8

2

32

10

2

8

01

23

2

8

01

23

Full PivotingFull Pivoting

• Full pivoting more stable, but only Full pivoting more stable, but only slightlyslightly

2

8

01

23

2

8

01

23

32

38

32

32

0

1

32

38

32

32

0

1

1

2

10

01

1

2

10

01

** Swap results Swap results 1 and 2 1 and 2

Operation CountOperation Count

• For one R.H.S., how many operations?For one R.H.S., how many operations?

• For each of n rows:For each of n rows:– Do n times:Do n times:

• For each of n+1 columns:For each of n+1 columns:

– One add, one multiplyOne add, one multiply

• Total = nTotal = n33+n+n22 multiplies, same # of adds multiplies, same # of adds

• Asymptotic behavior: when n is large, Asymptotic behavior: when n is large, dominated by ndominated by n33

Faster AlgorithmsFaster Algorithms

• Our goal is an algorithm that does this Our goal is an algorithm that does this inin11//33 nn33 operations, and does not require operations, and does not require

all R.H.S. to be known at beginningall R.H.S. to be known at beginning

• Before we see that, let’s look at a fewBefore we see that, let’s look at a fewspecial cases that are even fasterspecial cases that are even faster

Tridiagonal SystemsTridiagonal Systems

• Common special case:Common special case:

• Only main diagonal + 1 above and 1 Only main diagonal + 1 above and 1 belowbelow

4

3

2

1

4443

343332

232221

1211

00

0

0

00

b

b

b

b

aa

aaa

aaa

aa

4

3

2

1

4443

343332

232221

1211

00

0

0

00

b

b

b

b

aa

aaa

aaa

aa

Solving Tridiagonal SystemsSolving Tridiagonal Systems

• When solving using Gauss-Jordan:When solving using Gauss-Jordan:– Constant # of multiplies/adds in each rowConstant # of multiplies/adds in each row

– Each row only affects 2 othersEach row only affects 2 others

4

3

2

1

4443

343332

232221

1211

00

0

0

00

b

b

b

b

aa

aaa

aaa

aa

4

3

2

1

4443

343332

232221

1211

00

0

0

00

b

b

b

b

aa

aaa

aaa

aa

Running TimeRunning Time

• 2n loops, 4 multiply/adds per loop2n loops, 4 multiply/adds per loop(assuming correct bookkeeping)(assuming correct bookkeeping)

• This running time has a fundamentally This running time has a fundamentally different dependence on n: linear different dependence on n: linear instead of cubicinstead of cubic– Can say that tridiagonal algorithm is O(n) Can say that tridiagonal algorithm is O(n)

whilewhileGauss-Jordan is O(nGauss-Jordan is O(n33))

Big-O NotationBig-O Notation

• Informally, O(nInformally, O(n33) means that the dominant ) means that the dominant term for large n is cubicterm for large n is cubic

• More precisely, there exist a c and nMore precisely, there exist a c and n00 such such

that that running time running time c n c n33

ifif n > n n > n00

• This type of This type of asymptotic analysisasymptotic analysis is often is often usedusedto characterize different algorithmsto characterize different algorithms

Triangular SystemsTriangular Systems

• Another special case: A is lower-Another special case: A is lower-triangulartriangular

4

3

2

1

44434241

333231

2221

11

0

00

000

b

b

b

b

aaaa

aaa

aa

a

4

3

2

1

44434241

333231

2221

11

0

00

000

b

b

b

b

aaaa

aaa

aa

a


• Solve by forward substitutionSolve by forward substitution

4

3

2

1

44434241

333231

2221

11

0

00

000

b

b

b

b

aaaa

aaa

aa

a

4

3

2

1

44434241

333231

2221

11

0

00

000

b

b

b

b

aaaa

aaa

aa

a

11

11 a

bx

11

11 a

bx



4

3

2

1

44434241

333231

2221

11

0

00

000

b

b

b

b

aaaa

aaa

aa

a

4

3

2

1

44434241

333231

2221

11

0

00

000

b

b

b

b

aaaa

aaa

aa

a

22

12122 a

xabx

22

12122 a

xabx



4

3

2

1

44434241

333231

2221

11

0

00

000

b

b

b

b

aaaa

aaa

aa

a

4

3

2

1

44434241

333231

2221

11

0

00

000

b

b

b

b

aaaa

aaa

aa

a

33

23213133 a

xaxabx

33

23213133 a

xaxabx


• If A is upper triangular, solve by If A is upper triangular, solve by backsubstitutionbacksubstitution

5

4

3

2

1

55

4544

353433

25242322

1514131211

0000

000

00

0

b

b

b

b

b

a

aa

aaa

aaaa

aaaaa

5

4

3

2

1

55

4544

353433

25242322

1514131211

0000

000

00

0

b

b

b

b

b

a

aa

aaa

aaaa

aaaaa

55

55 a

bx

55

55 a

bx


• If A is upper triangular, solve by If A is upper triangular, solve by backsubstitutionbacksubstitution

5

4

3

2

1

55

4544

353433

25242322

1514131211

0000

000

00

0

b

b

b

b

b

a

aa

aaa

aaaa

aaaaa

5

4

3

2

1

55

4544

353433

25242322

1514131211

0000

000

00

0

b

b

b

b

b

a

aa

aaa

aaaa

aaaaa

44

54544 a

xabx

44

54544 a

xabx


• Both of these special cases can be solved Both of these special cases can be solved ininO(nO(n22) time) time

• This motivates a factorization approach to This motivates a factorization approach to solving arbitrary systems:solving arbitrary systems:– Find a way of writing A as LU, where L and U are Find a way of writing A as LU, where L and U are

both triangularboth triangular

– Ax=b Ax=b LUx=b LUx=b Ly=b Ly=b Ux=y Ux=y

– Time for factoring matrix dominates Time for factoring matrix dominates computationcomputation

Cholesky DecompositionCholesky Decomposition

• For symmetric matrices, choose U=LFor symmetric matrices, choose U=LTT

• Perform decompositionPerform decomposition

• Ax=b Ax=b LL LLTTx=b x=b Ly=b Ly=b LLTTx=yx=y

33

3222

312111

333231

2221

11

332313

232212

131211

00

00

00

l

ll

lll

lll

ll

l

aaa

aaa

aaa

33

3222

312111

333231

2221

11

332313

232212

131211

00

00

00

l

ll

lll

lll

ll

l

aaa

aaa

aaa


22

312123322332223121

221222222

222

221

11

1331133111

11

1221122111

1111112

11

l

llalallll

lalall

l

alall

l

alall

alal

22

312123322332223121

221222222

222

221

11

1331133111

11

1221122111

1111112

11

l

llalallll

lalall

l

alall

l

alall

alal

33

3222

312111

333231

2221

11

332313

232212

131211

00

00

00

l

ll

lll

lll

ll

l

aaa

aaa

aaa

33

3222

312111

333231

2221

11

332313

232212

131211

00

00

00

l

ll

lll

lll

ll

l

aaa

aaa

aaa


ii

i

kjkikij

ji

i

kikiiii

l

llal

lal

1

1

1

1

2

ii

i

kjkikij

ji

i

kikiiii

l

llal

lal

1

1

1

1

2

33

3222

312111

333231

2221

11

332313

232212

131211

00

00

00

l

ll

lll

lll

ll

l

aaa

aaa

aaa

33

3222

312111

333231

2221

11

332313

232212

131211

00

00

00

l

ll

lll

lll

ll

l

aaa

aaa

aaa


• This fails if it requires taking square root This fails if it requires taking square root of a negative numberof a negative number

• Need another condition on A: positive Need another condition on A: positive definitedefinite

For any v, vFor any v, vT T A v > 0A v > 0

(Equivalently, all positive eigenvalues)(Equivalently, all positive eigenvalues)


• Running time turns out to be Running time turns out to be 11//66nn33

– Still cubic, but much lower constantStill cubic, but much lower constant

• Result: this is preferred method for Result: this is preferred method for solvingsolvingsymmetric positive definite systemssymmetric positive definite systems

LU DecompositionLU Decomposition

• Again, factor A into LU, whereAgain, factor A into LU, whereL is lower triangular and U is upper L is lower triangular and U is upper triangulartriangular

• Last 2 steps in O(nLast 2 steps in O(n22) time, so total time ) time, so total time dominated by decompositiondominated by decomposition

Ax=bAx=bLUx=bLUx=bLy=bLy=bUx=yUx=y

Crout’s MethodCrout’s Method

• More unknowns than equations!More unknowns than equations!

• Let all lLet all liiii=1=1

33

2322

131211

333231

2221

11

333231

232221

131211

00

00

00

u

uu

uuu

lll

ll

l

aaa

aaa

aaa

33

2322

131211

333231

2221

11

333231

232221

131211

00

00

00

u

uu

uuu

lll

ll

l

aaa

aaa

aaa


33

2322

131211

3231

21

333231

232221

131211

00

0

1

01

001

u

uu

uuu

ll

l

aaa

aaa

aaa

33

2322

131211

3231

21

333231

232221

131211

00

0

1

01

001

u

uu

uuu

ll

l

aaa

aaa

aaa

22

123132323222321231

1221222222221221

1212

11

3131311131

11

2121211121

1111

u

ulalaulul

ulauauul

au

u

alaul

u

alaul

au

22

123132323222321231

1221222222221221

1212

11

3131311131

11

2121211121

1111

u

ulalaulul

ulauauul

au

u

alaul

u

alaul

au


• For i = 1..nFor i = 1..n– For j = 1..iFor j = 1..i

– For j = i+1..nFor j = i+1..n

33

2322

131211

3231

21

333231

232221

131211

00

0

1

01

001

u

uu

uuu

ll

l

aaa

aaa

aaa

33

2322

131211

3231

21

333231

232221

131211

00

0

1

01

001

u

uu

uuu

ll

l

aaa

aaa

aaa

1

1

j

kkijkjiji ulau

1

1

j

kkijkjiji ulau

ii

i

kkijkji

ji u

ulal

1

1

ii

i

kkijkji

ji u

ulal

1

1


• Interesting note: # of outputs = # of inputs,Interesting note: # of outputs = # of inputs,algorithm only refers to elements not output algorithm only refers to elements not output yetyet– Can do this in-place!Can do this in-place!

– Algorithm replaces A with matrixAlgorithm replaces A with matrixof l and u values, 1s are impliedof l and u values, 1s are implied

– Resulting matrix must be interpreted in a special Resulting matrix must be interpreted in a special way: not a regular matrixway: not a regular matrix

– Can rewrite forward/backsubstitution routines to Can rewrite forward/backsubstitution routines to use this “packed” l-u matrixuse this “packed” l-u matrix

333231

232221

131211

ull

uul

uuu

333231

232221

131211

ull

uul

uuu

LU DecompositionLU Decomposition

• Running time is Running time is 11//33nn33

– Only a factor of 2 slower than symmetric Only a factor of 2 slower than symmetric casecase

– This is the preferred general method forThis is the preferred general method forsolving linear equationssolving linear equations

• Pivoting very importantPivoting very important– Partial pivoting is sufficient, and widely Partial pivoting is sufficient, and widely

implementedimplemented

linear systems

Documents