steepest decent and conjugate gradients (cg). solving of the linear equation system
TRANSCRIPT
Steepest Decent and Conjugate Gradients (CG)
Steepest Decent and Conjugate Gradients (CG)
• Solving of the linear equation system bAx
Steepest Decent and Conjugate Gradients (CG)
• Solving of the linear equation system
• Problem: dimension n too big, or not enough time for gauss elimination
Iterative methods are used to get an approximate solution.
bAx
Steepest Decent and Conjugate Gradients (CG)
• Solving of the linear equation system
• Problem: dimension n too big, or not enough time for gauss elimination
Iterative methods are used to get an approximate solution.
• Definition Iterative method: given starting point , do steps
hopefully converge to the right solution
bAx
0x,, 21 xx
x
starting issues
starting issues
• Solving is equivalent to minimizing bAx cxbAxxxf TT
2
1:)(
starting issues
• Solving is equivalent to minimizing
• A has to be symmetric positive definite:
bAx cxbAxxxf TT
2
1:)(
00 xAxxAA TT
starting issues
• 02
1
2
1)(
!
bAxbAxxAxfsymmetricA
T
starting issues
•
• If A is also positive definite the solution of is the minimum
02
1
2
1)(
!
bAxbAxxAxfsymmetricA
T
bAx
starting issues
•
• If A is also positive definite the solution of is the minimum
02
1
2
1)(
!
bAxbAxxAxfsymmetricA
T
bAx
00
11
2
1
2
1)(
d
TT AddcbAbdbAf
starting issues
• error:
The norm of the error shows how far we are away from the exact solution, but can’t be computed without knowing of the exact solution .
xxe ii :
x
starting issues
• error:
The norm of the error shows how far we are away from the exact solution, but can’t be computed without knowing of the exact solution .
• residual:
can be calculated
xxe ii :
x)(: xfAeAxbr iii
Steepest Decent
Steepest Decent
• We are at the point . How do we reach ?ix 1ix
Steepest Decent
• We are at the point . How do we reach ?
• Idea: go into the direction in which decreases most quickly ( )
ix 1ix
)(xf
ii rxf )(
Steepest Decent
• We are at the point . How do we reach ?
• Idea: go into the direction in which decreases most quickly ( )
• how far should we go?
ix 1ix
)(xf
ii rxf )(
Steepest Decent
• We are at the point . How do we reach ?
• Idea: go into the direction in which decreases most quickly ( )
• how far should we go?
Choose so that is minimized:
ix 1ix
)(xf
ii rxf )(
)( ii rxf
Steepest Decent
• We are at the point . How do we reach ?
• Idea: go into the direction in which decreases most quickly ( )
• how far should we go?
Choose so that is minimized:
ix 1ix
)(xf
ii rxf )(
)( ii rxf
0)( ii rxfd
d
Steepest Decent
• We are at the point . How do we reach ?
• Idea: go into the direction in which decreases most quickly ( )
• how far should we go?
Choose so that is minimized:
ix 1ix
)(xf
ii rxf )(
)( ii rxf
0)( ii rxfd
d
0)( iT
ii rrxf
Steepest Decent
• We are at the point . How do we reach ?
• Idea: go into the direction in which decreases most quickly ( )
• how far should we go?
Choose so that is minimized:
ix 1ix
)(xf
ii rxf )(
)( ii rxf
0)( ii rxfd
d
0)( iT
ii rrxf
0))(( iT
ii rbrxA
Steepest Decent
• We are at the point . How do we reach ?
• Idea: go into the direction in which decreases most quickly ( )
• how far should we go?
Choose so that is minimized:
ix 1ix
)(xf
ii rxf )(
)( ii rxf
0)( ii rxfd
d
0)( iT
ii rrxf
0))(( iT
ii rbrxA
iT
r
iiT
i rAxbrAr
i
)()(
Steepest Decent
• We are at the point . How do we reach ?
• Idea: go into the direction in which decreases most quickly ( )
• how far should we go?
Choose so that is minimized:
ix 1ix
)(xf
ii rxf )(
)( ii rxf
0)( ii rxfd
d
0)( iT
ii rrxf
0))(( iT
ii rbrxA
iT
r
iiT
i rAxbrAr
i
)()( i
Ti
iTi
Arr
rr
Steepest Decent
one step of steepest decent can be calculated as follows:
iiii
Ti
iTi
i
ii
rxx
Arr
rr
Axbr
1
Steepest Decent
one step of steepest decent can be calculated as follows:
• stopping criterion: or with an given small
It would be better to use the error instead of the residual, but you can’t calculate the error.
iiii
Ti
iTi
i
ii
rxx
Arr
rr
Axbr
1
maxii 0rri
Steepest Decent
Method of steepest decent:
1
)(
0
00max
0
0
ii
Axbr
rxxArr
rr
rrrrandiiwhile
rr
Axbr
i
T
T
TT
Steepest Decent
• As you can see the starting point is important!
Steepest Decent
• As you can see the starting point is important!
When you know anything about the solution use it to guess a good starting point. Otherwise you can choose a starting point you want e.g. .00 x
Steepest Decent - Convergence
Steepest Decent - Convergence
• Definition energy norm: Axxx T
A:
Steepest Decent - Convergence
• Definition energy norm:
• Definition condition:
( is the largest and the smallest eigenvalue of A)
Axxx T
A:
min
max:
max min
Steepest Decent - Convergence
• Definition energy norm:
• Definition condition:
( is the largest and the smallest eigenvalue of A)
•
convergence gets worse when the condition gets larger
Axxx T
A:
min
max:
max min
A
i
Aiee 01
1
Conjugate Gradients
Conjugate Gradients
• is there a better direction?
Conjugate Gradients
• is there a better direction?
• Idea: orthogonal search directions110 ,,, nddd
Conjugate Gradients
• is there a better direction?
• Idea: orthogonal search directions110 ,,, nddd
1
0
n
iiidx
Conjugate Gradients
• is there a better direction?
• Idea: orthogonal search directions
• only walk once in each direction and minimize
110 ,,, nddd
1
0
n
iiidx
Conjugate Gradients
• is there a better direction?
• Idea: orthogonal search directions
• only walk once in each direction and minimize
maximal n steps are needed to reach the exact solution
110 ,,, nddd
1
0
n
iiidx
Conjugate Gradients
• is there a better direction?
• Idea: orthogonal search directions
• only walk once in each direction and minimize
maximal n steps are needed to reach the exact solution
has to be orthogonal to
110 ,,, nddd
1
0
n
iiidx
1 ie id
Conjugate Gradients
• example with the coordinate axes as orthogonal search directions:
Conjugate Gradients
• example with the coordinate axes as orthogonal search directions:
Problem: can’t be computed
because
(you don’t know !)
iTi
iTi
idd
ed
ie
Conjugate Gradients
• new idea: A-orthogonal110 ,,, nddd
Conjugate Gradients
• new idea: A-orthogonal
• Definition A-orthogonal: A-orthogonal
(reminder: orthogonal: )
110 ,,, nddd
ji dd , 0 jTi Add
ji dd , 0 jTi dd
Conjugate Gradients
• new idea: A-orthogonal
• Definition A-orthogonal: A-orthogonal
(reminder: orthogonal: )
• now has to be A-orthogonal to
110 ,,, nddd
ji dd , 0 jTi Add
ji dd , 0 jTi dd
1ie id
Conjugate Gradients
• new idea: A-orthogonal
• Definition A-orthogonal: A-orthogonal
(reminder: orthogonal: )
• now has to be A-orthogonal to
110 ,,, nddd
ji dd , 0 jTi Add
ji dd , 0 jTi dd
1ie id
iTi
iTi
iTi
iTi
i Add
rd
Add
Aed
Conjugate Gradients
• new idea: A-orthogonal
• Definition A-orthogonal: A-orthogonal
(reminder: orthogonal: )
• now has to be A-orthogonal to
can be computed!
110 ,,, nddd
ji dd , 0 jTi Add
ji dd , 0 jTi dd
1ie id
iTi
iTi
iTi
iTi
i Add
rd
Add
Aed
Conjugate Gradients
• A set of A-orthogonal directions can be found with n linearly independent vectors and conjugate Gram-Schmidt (same idea as Gram-Schmidt).
iu
Conjugate Gradients
• Gram-Schmidt:
linearly independent vectors10 ,, nuu
Conjugate Gradients
• Gram-Schmidt:
linearly independent vectors10 ,, nuu
jTj
jTi
ij
i
jjijii
dd
du
dudi
ud
1
0
00
:0
Conjugate Gradients
• Gram-Schmidt:
linearly independent vectors
• conjugate Gram-Schmidt:
10 ,, nuu
jTj
jTi
ij Add
Adu
jTj
jTi
ij
i
jjijii
dd
du
dudi
ud
1
0
00
:0
Conjugate Gradients
• A set of A-orthogonal directions can be found with n linearly independent vectors and conjugate Gram-Schmidt (same idea as Gram-Schmidt).
• CG works by setting (makes conjugate Gram-Schmidt easy)
iu
ii ru
Conjugate Gradients
• A set of A-orthogonal directions can be found with n linearly independent vectors and conjugate Gram-Schmidt (same idea as Gram-Schmidt).
• CG works by setting (makes conjugate Gram-Schmidt easy)
with1 iiii drd 11
i
Ti
iTi
i rr
rr
ii ru
iu
Conjugate Gradients
• 0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
Conjugate Gradients
•
•
0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
1
0
i
kkikii dud
Conjugate Gradients
•
•
0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
1
0
i
kkikii dud
1
00
0:i
kjk
jTkikj
Tij
Ti rdrurdji
Conjugate Gradients
•
•
0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
1
0
i
kkikii dud
1
00
0:i
kjk
jTkikj
Tij
Ti rdrurdji
jiru jTi 0
Conjugate Gradients
•
•
0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
1
0
i
kkikii dud
1
00
0:i
kjk
jTkikj
Tij
Ti rdrurdji
jiru jTi 0
ijjTi
jTiii
rr
jirrru
0:
Conjugate Gradients
•
•
•
0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
ijjTi
jTiii
rr
jirrru
0:
iTi
i
kjk
jTkiki
Tii
Ti rurdrurd
1
00
Conjugate Gradients
• jiAdd
Adr
jTj
jTi
ij
Conjugate Gradients
•
•
jiAdd
Adr
jTj
jTi
ij
jjjjjjjj AdrdeAAer )(11
Conjugate Gradients
•
•
jiAdd
Adr
jTj
jTi
ij
jjjjjjjj AdrdeAAer )(11
jTijj
Tij
Ti Adrrrrr 1
Conjugate Gradients
•
•
jiAdd
Adr
jTj
jTi
ij
jjjjjjjj AdrdeAAer )(11
jTijj
Tij
Ti Adrrrrr 1
1 jTij
Tij
Tij rrrrAdr
Conjugate Gradients
1 jTij
Tij
Tij rrrrAdr
10
11
jiji
jirr
jirr
Adr
rr
i
iTi
i
iTi
jTi
ijjTi
Conjugate Gradients
10 ji
ij
Conjugate Gradients
1
10
1111111
jirr
rr
rd
rr
Add
rrji
iTi
iTi
iTi
iTi
def
iTii
iTiij
Method of Conjugate Gradients:
00
0
i
rr
rd
Axbr
1
)( 00max
ii
drd
rr
rr
Axbr
rr
dxxAdd
rr
rrrrandiiwhile
oldTold
T
old
T
T
TT
Conjugate Gradients - Convergence
Conjugate Gradients - Convergence
• A
i
Aiee 0
1
12
Conjugate Gradients - Convergence
•
• for steepest decent for CG
Convergence of CG is much better!
A
i
Aiee 0
1
12