1 chapter 6 general strategy for gradient methods (1) calculate a search direction (2) select a step...
TRANSCRIPT
1
Ch
apte
r 6
General Strategy for Gradient methods
(1) Calculate a search direction(2) Select a step length in that direction to reduce f(x)
1k k k k kx x s x x
Steepest DescentSearch Direction
( ) k ks f x Don’t need to normalize
Method terminates at any stationary point. Why?
0)( xf
ks
2
Ch
apte
r 6
3
Ch
apte
r 6
4
Ch
apte
r 6
Numerical MethodUse coarse search first (1) Fixed ( = 1) or variable ( = 1, 2, ½, etc.)
Options for optimizing (1) Use interpolation such as quadratic, cubic(2) Region Elimination (Golden Search)(3) Newton, Secant, Quasi-Newton(4) Random(5) Analytical optimization
(1), (3), and (5) are preferred. However, it maynot be desirable to exactly optimize (better togenerate new search directions).
5
Ch
apte
r 6
6
Ch
apte
r 6
7
Conjugate Search Directions
• Improvement over gradient method for general quadratic functions
• Basis for many NLP techniques• Two search directions are conjugate relative to Q if
• To minimize f(xnx1) when H is a constant matrix (=Q), you are guaranteed to reach the optimum in n conjugate direction stages if you minimize exactly at each stage
(one-dimensional search)
( ) ( ) 0i T j s Q s
Ch
apte
r 6
8
Ch
apte
r 6
9
Conjugate Gradient Method0 0
0 0
At calculate ( ). Let
( )
f
f
Step 1. x x
s x
0
1 0 0 0
Save ( ) and compute
f
Step 2. x
x x s
by minimizing f(x) with respect to in the s0 direction (i.e., carry out a unidimensional search for 0).
Step 3. Calculate The new search direction is a linear combination of
For the kth iteration the relation is
For a quadratic function it can be shown that these successive search directions are conjugate.After n iterations (k = n), the quadratic function is minimized. For a nonquadratic function,the procedure cycles again with xn+1 becoming x0.
Step 4. Test for convergence to the minimum of f(x). If convergence is not attained, return to step 3.
Step n. Terminate the algorithm when is less than some prescribed tolerance.
1 11 1 0
0 0
( ) ( )( )
( ) ( )
T
T
f ff
f f
x x
s x sx x
1 11 1 ( ) ( )
( )( ) ( )
T k kk k k
T k k
f ff
f f
x xs x s
x x
1 1( ), ( ).f fx x
( )kf x
(6.6)
0 1 and ( ) :fs x
Ch
apte
r 6