5/20/2015 1 multidimensional gradient methods in optimization major: all engineering majors...
TRANSCRIPT
![Page 1: 5/20/2015 1 Multidimensional Gradient Methods in Optimization Major: All Engineering Majors Authors: Autar Kaw, Ali](https://reader033.vdocument.in/reader033/viewer/2022042608/56649d0b5503460f949de8c4/html5/thumbnails/1.jpg)
04/18/23http://
numericalmethods.eng.usf.edu 1
Multidimensional Gradient Methods in Optimization
Major: All Engineering Majors
Authors: Autar Kaw, Ali Yalcin
http://numericalmethods.eng.usf.eduTransforming Numerical Methods Education for STEM
Undergraduates
![Page 2: 5/20/2015 1 Multidimensional Gradient Methods in Optimization Major: All Engineering Majors Authors: Autar Kaw, Ali](https://reader033.vdocument.in/reader033/viewer/2022042608/56649d0b5503460f949de8c4/html5/thumbnails/2.jpg)
Steepest Ascent/Descent Method
http://numericalmethods.eng.usf.edu
![Page 3: 5/20/2015 1 Multidimensional Gradient Methods in Optimization Major: All Engineering Majors Authors: Autar Kaw, Ali](https://reader033.vdocument.in/reader033/viewer/2022042608/56649d0b5503460f949de8c4/html5/thumbnails/3.jpg)
Multidimensional Gradient Methods -Overview
Use information from the derivatives of the optimization function to guide the search
Finds solutions quicker compared with direct search methods
A good initial estimate of the solution is required
The objective function needs to be differentiable
http://numericalmethods.eng.usf.edu 3
![Page 4: 5/20/2015 1 Multidimensional Gradient Methods in Optimization Major: All Engineering Majors Authors: Autar Kaw, Ali](https://reader033.vdocument.in/reader033/viewer/2022042608/56649d0b5503460f949de8c4/html5/thumbnails/4.jpg)
Gradients The gradient is a vector operator denoted
by (referred to as “del”) When applied to a function , it represents
the functions directional derivatives The gradient is the special case where
the direction of the gradient is the direction of most or the steepest ascent/descent
The gradient is calculated by
http://numericalmethods.eng.usf.edu 4
jiy
f
x
ff
![Page 5: 5/20/2015 1 Multidimensional Gradient Methods in Optimization Major: All Engineering Majors Authors: Autar Kaw, Ali](https://reader033.vdocument.in/reader033/viewer/2022042608/56649d0b5503460f949de8c4/html5/thumbnails/5.jpg)
Gradients-ExampleCalculate the gradient to determine the direction
of the steepest slope at point (2, 1) for the function
Solution: To calculate the gradient we would need to calculate
which are used to determine the gradient at point (2,1) as
http://numericalmethods.eng.usf.edu 5
22, yxyxf
4)1)(2(22 22
xyx
f 8)1()2(22 22
yxy
f
j8i4 f
![Page 6: 5/20/2015 1 Multidimensional Gradient Methods in Optimization Major: All Engineering Majors Authors: Autar Kaw, Ali](https://reader033.vdocument.in/reader033/viewer/2022042608/56649d0b5503460f949de8c4/html5/thumbnails/6.jpg)
Hessians The Hessian matrix or just the Hessian is
the Jacobian matrix of second-order partial derivatives of a function.
The determinant of the Hessian matrix is also referred to as the Hessian.
For a two dimensional function the Hessian matrix is simply
http://numericalmethods.eng.usf.edu 6
2
22
2
2
2
y
f
xy
fyx
f
x
f
H
![Page 7: 5/20/2015 1 Multidimensional Gradient Methods in Optimization Major: All Engineering Majors Authors: Autar Kaw, Ali](https://reader033.vdocument.in/reader033/viewer/2022042608/56649d0b5503460f949de8c4/html5/thumbnails/7.jpg)
Hessians cont.The determinant of the Hessian
matrix denoted by can have three cases:
1. If and then has a local minimum.
2. If and then has a local maximum.
3. If then has a saddle point.
http://numericalmethods.eng.usf.edu 7
H0H 0/ 222 xf yxf ,
0H 0/ 222 xf yxf ,
0H yxf ,
![Page 8: 5/20/2015 1 Multidimensional Gradient Methods in Optimization Major: All Engineering Majors Authors: Autar Kaw, Ali](https://reader033.vdocument.in/reader033/viewer/2022042608/56649d0b5503460f949de8c4/html5/thumbnails/8.jpg)
Hessians-ExampleCalculate the hessian matrix at point (2, 1) for
the function Solution: To calculate the Hessian matrix; the
partial derivatives must be evaluated as
resulting in the Hessian matrix
http://numericalmethods.eng.usf.edu 8
22, yxyxf
4)1(22 2222
2
yx
f8)2(22 22
2
2
xy
f 8)1)(2(4422
xyxy
f
yx
f
88
84
2
22
2
2
2
y
f
xy
fyx
f
x
f
H
![Page 9: 5/20/2015 1 Multidimensional Gradient Methods in Optimization Major: All Engineering Majors Authors: Autar Kaw, Ali](https://reader033.vdocument.in/reader033/viewer/2022042608/56649d0b5503460f949de8c4/html5/thumbnails/9.jpg)
http://numericalmethods.eng.usf.edu9
Steepest Ascent/Descent Method
Starts from an initial point and looks for a local optimal solution along a gradient.
The gradient at the initial solution is calculated.
A new solution is found at the local optimum along the gradient
The subsequent iterations involve using the local optima along the new gradient as the initial point.
![Page 10: 5/20/2015 1 Multidimensional Gradient Methods in Optimization Major: All Engineering Majors Authors: Autar Kaw, Ali](https://reader033.vdocument.in/reader033/viewer/2022042608/56649d0b5503460f949de8c4/html5/thumbnails/10.jpg)
Example
Determine the minimum of the function
Use the point (2,1) as the initial estimate of the optimal solution.
http://numericalmethods.eng.usf.edu10
42, 22 xyxyxf
![Page 11: 5/20/2015 1 Multidimensional Gradient Methods in Optimization Major: All Engineering Majors Authors: Autar Kaw, Ali](https://reader033.vdocument.in/reader033/viewer/2022042608/56649d0b5503460f949de8c4/html5/thumbnails/11.jpg)
http://numericalmethods.eng.usf.edu11
SolutionIteration 1: To calculate the gradient; the partial derivatives must be evaluated as
Now the function can be expressed along the direction of gradient as
42)2(222
xx
f 2)1(22
yy
f
j2i4 f
yxf ,
4)42(2)21()42()21,42(, 2200
hhhhhfhy
fyh
x
fxf
132820)( 2 hhhg
![Page 12: 5/20/2015 1 Multidimensional Gradient Methods in Optimization Major: All Engineering Majors Authors: Autar Kaw, Ali](https://reader033.vdocument.in/reader033/viewer/2022042608/56649d0b5503460f949de8c4/html5/thumbnails/12.jpg)
http://numericalmethods.eng.usf.edu12
Solution Cont.Iteration 1 continued: This is a simple function and it is easy to determine by taking the first derivative and solving for its roots.
This means that traveling a step size of along the gradient reaches a minimum value for the function in this direction. These values are substituted back to calculate a new value for x and y as follows:
7.0* h
7.0h
4.0)7.0(21
8.0)7.0(42
y
x
2.34.0,8.0 f 131,2 fNote that
![Page 13: 5/20/2015 1 Multidimensional Gradient Methods in Optimization Major: All Engineering Majors Authors: Autar Kaw, Ali](https://reader033.vdocument.in/reader033/viewer/2022042608/56649d0b5503460f949de8c4/html5/thumbnails/13.jpg)
http://numericalmethods.eng.usf.edu13
Solution Cont.Iteration 2: The new initial point is .We calculate the gradient at this point as
4.0,8.0
4.02)8.0(222
xx
f 8.0)4.0(22
yy
f
j8.0i4.0 f
4)4.08.0(2)8.04.0()4.08.0()8.04.0,4.08.0(, 2200
hhhhhfhy
fyh
x
fxf
2.38.08.0)( 2 hhhg 5.0* h
0)5.0(8.04.0
1)5.0(4.08.0
y
x
30,1 f 2.34.0,8.0 f
![Page 14: 5/20/2015 1 Multidimensional Gradient Methods in Optimization Major: All Engineering Majors Authors: Autar Kaw, Ali](https://reader033.vdocument.in/reader033/viewer/2022042608/56649d0b5503460f949de8c4/html5/thumbnails/14.jpg)
http://numericalmethods.eng.usf.edu14
Solution Cont.Iteration 3: The new initial point is .We calculate the gradient at this point as
0,1
02)1(222
xx
f 0)0(22
yy
f
j0i0 f
This indicates that the current location is a local optimum along this gradient and no improvement can be gained by moving in any direction. The minimum of the function is at point (-1,0).
![Page 15: 5/20/2015 1 Multidimensional Gradient Methods in Optimization Major: All Engineering Majors Authors: Autar Kaw, Ali](https://reader033.vdocument.in/reader033/viewer/2022042608/56649d0b5503460f949de8c4/html5/thumbnails/15.jpg)
Additional ResourcesFor all resources on this topic such as digital audiovisual lectures, primers, textbook chapters, multiple-choice tests, worksheets in MATLAB, MATHEMATICA, MathCad and MAPLE, blogs, related physical problems, please visit
http://nm.mathforcollege.com/topics/opt_multidimensional_gradient.html