steepest decent and conjugate gradients (cg)
DESCRIPTION
Steepest Decent and Conjugate Gradients (CG). Steepest Decent and Conjugate Gradients (CG). Solving of the linear equation system. Steepest Decent and Conjugate Gradients (CG). Solving of the linear equation system Problem : dimension n too big, or not enough time for gauss elimination - PowerPoint PPT PresentationTRANSCRIPT
Steepest Decent and Conjugate Gradients (CG)
Steepest Decent and Conjugate Gradients (CG)
• Solving of the linear equation system bAx
Steepest Decent and Conjugate Gradients (CG)
• Solving of the linear equation system
• Problem: dimension n too big, or not enough time for gauss elimination
Iterative methods are used to get an approximate solution.
bAx
Steepest Decent and Conjugate Gradients (CG)
• Solving of the linear equation system
• Problem: dimension n too big, or not enough time for gauss elimination
Iterative methods are used to get an approximate solution.
• Definition Iterative method: given starting point , do steps
hopefully converge to the right solution
bAx
0x,, 21 xx
x
starting issues
starting issues
• Solving is equivalent to minimizing bAx cxbAxxxf TT
2
1:)(
starting issues
• Solving is equivalent to minimizing
• A has to be symmetric positive definite:
bAx cxbAxxxf TT
2
1:)(
00 xAxxAA TT
starting issues
• 02
1
2
1)(
!
bAxbAxxAxfsymmetricA
T
starting issues
•
• If A is also positive definite the solution of is the minimum
02
1
2
1)(
!
bAxbAxxAxfsymmetricA
T
bAx
starting issues
•
• If A is also positive definite the solution of is the minimum
02
1
2
1)(
!
bAxbAxxAxfsymmetricA
T
bAx
00
11
2
1
2
1)(
d
TT AddcbAbdbAf
starting issues
• error:
The norm of the error shows how far we are away from the exact solution, but can’t be computed without knowing of the exact solution .
xxe ii :
x
starting issues
• error:
The norm of the error shows how far we are away from the exact solution, but can’t be computed without knowing of the exact solution .
• residual:
can be calculated
xxe ii :
x)(: xfAeAxbr iii
Steepest Decent
Steepest Decent
• We are at the point . How do we reach ?ix 1ix
Steepest Decent
• We are at the point . How do we reach ?
• Idea: go into the direction in which decreases most quickly ( )
ix 1ix
)(xf
ii rxf )(
Steepest Decent
• We are at the point . How do we reach ?
• Idea: go into the direction in which decreases most quickly ( )
• how far should we go?
ix 1ix
)(xf
ii rxf )(
Steepest Decent
• We are at the point . How do we reach ?
• Idea: go into the direction in which decreases most quickly ( )
• how far should we go?
Choose so that is minimized:
ix 1ix
)(xf
ii rxf )(
)( ii rxf
Steepest Decent
• We are at the point . How do we reach ?
• Idea: go into the direction in which decreases most quickly ( )
• how far should we go?
Choose so that is minimized:
ix 1ix
)(xf
ii rxf )(
)( ii rxf
0)( ii rxfd
d
Steepest Decent
• We are at the point . How do we reach ?
• Idea: go into the direction in which decreases most quickly ( )
• how far should we go?
Choose so that is minimized:
ix 1ix
)(xf
ii rxf )(
)( ii rxf
0)( ii rxfd
d
0)( iT
ii rrxf
Steepest Decent
• We are at the point . How do we reach ?
• Idea: go into the direction in which decreases most quickly ( )
• how far should we go?
Choose so that is minimized:
ix 1ix
)(xf
ii rxf )(
)( ii rxf
0)( ii rxfd
d
0)( iT
ii rrxf
0))(( iT
ii rbrxA
Steepest Decent
• We are at the point . How do we reach ?
• Idea: go into the direction in which decreases most quickly ( )
• how far should we go?
Choose so that is minimized:
ix 1ix
)(xf
ii rxf )(
)( ii rxf
0)( ii rxfd
d
0)( iT
ii rrxf
0))(( iT
ii rbrxA
iT
r
iiT
i rAxbrAr
i
)()(
Steepest Decent
• We are at the point . How do we reach ?
• Idea: go into the direction in which decreases most quickly ( )
• how far should we go?
Choose so that is minimized:
ix 1ix
)(xf
ii rxf )(
)( ii rxf
0)( ii rxfd
d
0)( iT
ii rrxf
0))(( iT
ii rbrxA
iT
r
iiT
i rAxbrAr
i
)()( i
Ti
iTi
Arr
rr
Steepest Decent
one step of steepest decent can be calculated as follows:
iiii
Ti
iTi
i
ii
rxx
Arr
rr
Axbr
1
Steepest Decent
one step of steepest decent can be calculated as follows:
• stopping criterion: or with an given small
It would be better to use the error instead of the residual, but you can’t calculate the error.
iiii
Ti
iTi
i
ii
rxx
Arr
rr
Axbr
1
maxii 0rri
Steepest Decent
Method of steepest decent:
1
)(
0
00max
0
0
ii
Axbr
rxxArr
rr
rrrrandiiwhile
rr
Axbr
i
T
T
TT
Steepest Decent
• As you can see the starting point is important!
Steepest Decent
• As you can see the starting point is important!
When you know anything about the solution use it to guess a good starting point. Otherwise you can choose a starting point you want e.g. .00 x
Steepest Decent - Convergence
Steepest Decent - Convergence
• Definition energy norm: Axxx T
A:
Steepest Decent - Convergence
• Definition energy norm:
• Definition condition:
( is the largest and the smallest eigenvalue of A)
Axxx T
A:
min
max:
max min
Steepest Decent - Convergence
• Definition energy norm:
• Definition condition:
( is the largest and the smallest eigenvalue of A)
•
convergence gets worse when the condition gets larger
Axxx T
A:
min
max:
max min
A
i
Aiee 01
1
Conjugate Gradients
Conjugate Gradients
• is there a better direction?
Conjugate Gradients
• is there a better direction?
• Idea: orthogonal search directions110 ,,, nddd
Conjugate Gradients
• is there a better direction?
• Idea: orthogonal search directions110 ,,, nddd
1
0
n
iiidx
Conjugate Gradients
• is there a better direction?
• Idea: orthogonal search directions
• only walk once in each direction and minimize
110 ,,, nddd
1
0
n
iiidx
Conjugate Gradients
• is there a better direction?
• Idea: orthogonal search directions
• only walk once in each direction and minimize
maximal n steps are needed to reach the exact solution
110 ,,, nddd
1
0
n
iiidx
Conjugate Gradients
• is there a better direction?
• Idea: orthogonal search directions
• only walk once in each direction and minimize
maximal n steps are needed to reach the exact solution
has to be orthogonal to
110 ,,, nddd
1
0
n
iiidx
1 ie id
Conjugate Gradients
• example with the coordinate axes as orthogonal search directions:
Conjugate Gradients
• example with the coordinate axes as orthogonal search directions:
Problem: can’t be computed
because
(you don’t know !)
iTi
iTi
idd
ed
ie
Conjugate Gradients
• new idea: A-orthogonal110 ,,, nddd
Conjugate Gradients
• new idea: A-orthogonal
• Definition A-orthogonal: A-orthogonal
(reminder: orthogonal: )
110 ,,, nddd
ji dd , 0 jTi Add
ji dd , 0 jTi dd
Conjugate Gradients
• new idea: A-orthogonal
• Definition A-orthogonal: A-orthogonal
(reminder: orthogonal: )
• now has to be A-orthogonal to
110 ,,, nddd
ji dd , 0 jTi Add
ji dd , 0 jTi dd
1ie id
Conjugate Gradients
• new idea: A-orthogonal
• Definition A-orthogonal: A-orthogonal
(reminder: orthogonal: )
• now has to be A-orthogonal to
110 ,,, nddd
ji dd , 0 jTi Add
ji dd , 0 jTi dd
1ie id
iTi
iTi
iTi
iTi
i Add
rd
Add
Aed
Conjugate Gradients
• new idea: A-orthogonal
• Definition A-orthogonal: A-orthogonal
(reminder: orthogonal: )
• now has to be A-orthogonal to
can be computed!
110 ,,, nddd
ji dd , 0 jTi Add
ji dd , 0 jTi dd
1ie id
iTi
iTi
iTi
iTi
i Add
rd
Add
Aed
Conjugate Gradients
• A set of A-orthogonal directions can be found with n linearly independent vectors and conjugate Gram-Schmidt (same idea as Gram-Schmidt).
iu
Conjugate Gradients
• Gram-Schmidt:
linearly independent vectors10 ,, nuu
Conjugate Gradients
• Gram-Schmidt:
linearly independent vectors10 ,, nuu
jTj
jTi
ij
i
jjijii
dd
du
dudi
ud
1
0
00
:0
Conjugate Gradients
• Gram-Schmidt:
linearly independent vectors
• conjugate Gram-Schmidt:
10 ,, nuu
jTj
jTi
ij Add
Adu
jTj
jTi
ij
i
jjijii
dd
du
dudi
ud
1
0
00
:0
Conjugate Gradients
• A set of A-orthogonal directions can be found with n linearly independent vectors and conjugate Gram-Schmidt (same idea as Gram-Schmidt).
• CG works by setting (makes conjugate Gram-Schmidt easy)
iu
ii ru
Conjugate Gradients
• A set of A-orthogonal directions can be found with n linearly independent vectors and conjugate Gram-Schmidt (same idea as Gram-Schmidt).
• CG works by setting (makes conjugate Gram-Schmidt easy)
with1 iiii drd 11
i
Ti
iTi
i rr
rr
ii ru
iu
Conjugate Gradients
• 0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
Conjugate Gradients
•
•
0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
1
0
i
kkikii dud
Conjugate Gradients
•
•
0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
1
0
i
kkikii dud
1
00
0:i
kjk
jTkikj
Tij
Ti rdrurdji
Conjugate Gradients
•
•
0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
1
0
i
kkikii dud
1
00
0:i
kjk
jTkikj
Tij
Ti rdrurdji
jiru jTi 0
Conjugate Gradients
•
•
0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
1
0
i
kkikii dud
1
00
0:i
kjk
jTkikj
Tij
Ti rdrurdji
jiru jTi 0
ijjTi
jTiii
rr
jirrru
0:
Conjugate Gradients
•
•
•
0:1
0
1
n
jkk
Tik
n
jkkk
Tii
Tij
Ti AdddAdAedrdji
ijjTi
jTiii
rr
jirrru
0:
iTi
i
kjk
jTkiki
Tii
Ti rurdrurd
1
00
Conjugate Gradients
• jiAdd
Adr
jTj
jTi
ij
Conjugate Gradients
•
•
jiAdd
Adr
jTj
jTi
ij
jjjjjjjj AdrdeAAer )(11
Conjugate Gradients
•
•
jiAdd
Adr
jTj
jTi
ij
jjjjjjjj AdrdeAAer )(11
jTijj
Tij
Ti Adrrrrr 1
Conjugate Gradients
•
•
jiAdd
Adr
jTj
jTi
ij
jjjjjjjj AdrdeAAer )(11
jTijj
Tij
Ti Adrrrrr 1
1 jTij
Tij
Tij rrrrAdr
Conjugate Gradients
1 jTij
Tij
Tij rrrrAdr
10
11
jiji
jirr
jirr
Adr
rr
i
iTi
i
iTi
jTi
ijjTi
Conjugate Gradients
10 ji
ij
Conjugate Gradients
1
10
1111111
jirr
rr
rd
rr
Add
rrji
iTi
iTi
iTi
iTi
def
iTii
iTiij
Method of Conjugate Gradients:
00
0
i
rr
rd
Axbr
1
)( 00max
ii
drd
rr
rr
Axbr
rr
dxxAdd
rr
rrrrandiiwhile
oldTold
T
old
T
T
TT
Conjugate Gradients - Convergence
Conjugate Gradients - Convergence
• A
i
Aiee 0
1
12
Conjugate Gradients - Convergence
•
• for steepest decent for CG
Convergence of CG is much better!
A
i
Aiee 0
1
12