functions of a single variable - indian institute of ...shibayan/mcc 52105... · gradient of a...

Optimization Theory MMC 52212 / MME 52106

by

Dr. Shibayan Sarkar Department of Mechanical Engg. Indian School of Mines Dhanbad

Multivariable Optimization

[Unconstrained]

When? Multiple decision variable Gradient of a function is not a scalar quantity. It is a vector. Optimality criteria can be derived by using the definition of a local optimal point

and by using Taylor’s series expansion of a multivariable function. unidirectional method Direct search Gradient based method

The objective function of a N variable is represented as x1, x2 ….xN. The gradient vector is represented as

( )

( )

1 2

( ) , ......t

Tt

Nx

f f ff xx x x

∂ ∂ ∂∇ = ∂ ∂ ∂

( )

2 2 2

21 1 2 12 2 2

2 ( ) 21 2 2 2

2 2 2

21 2

......

......( )

...........

......t

T

N

tN

N N Nx

f f fx x x x x

f f ff x x x x x x

f f fx x x x x

∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂

∇ = ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂

Hessian Matrix A point is a stationary point if .

Furthermore the point is minimum , maximum and inflation point if is positive-definite, negative-definite or otherwise.

( ) 0f x∇ =x

2 ( ) 0f x∇ =

Unidirectional Method Find minimum point at a particular direction One dimensional search.

x1 x2 f consider 0 0 200 200 0 1 181 180 0 2 164 160 0 3 149 150 0 4 136 140 0 5 125 130 1 0 181 180 1 1 162 160 1 2 145 150 1 3 130 130 1 4 117 120 1 5 106 110 2 0 164 160 2 1 145 150 2 2 128 130 2 3 113 110 2 4 100 100 2 5 89 90 3 0 149 150 3 1 130 130 3 2 113 110 3 3 98 100 3 4 85 90 3 5 74 70 4 0 136 140 4 1 117 120 4 2 100 100 4 3 85 90 4 4 72 70 4 5 61 60 5 0 125 130 5 1 106 110 5 2 89 90 5 3 74 70 5 4 61 60 5 5 50 50

Contour line for 90 Contour line for 130

Unidirectional Method: Example 1

(1) ( )

( )

( ) t

tx x

sα α−

=

1 2

1 2

( ) (2,1) 0.5(2,5)

( ) (2) ( ) (1)0.5 0.5(2) (5)

( ) 3.0 ( ) 0.5( ) (3.0,3.5)

x

x xand

x and xx

α

α α

α αα

−=

− −= =

= ==

a

b

2

5

Graphical method

Unidirectional Method: Example 1

(1)

1 2

1 2

( ) (2,1) 0.5(2,5)

( ) (2) ( ) (1)0.5 0.5(2) (5)

( ) 3.0 ( ) 0.5( ) (3.0,3.5)

x

x xand

x and xx

α

α α

α αα

−=

− −= =

= ==

a

b

2

5

1 2

( ) (2,1) (2,5)( ) 2 2 ( ) 1 5

xx and xα αα α α α= += + = +

(1) (2)

From equitation (2) ….

(3)

Substitute the value of eq(1) by eq(3) …. 2 2 2( ) (2 2 10) (1 5 10) 29 122 145Min f α α α α α= + − + + − = − +

2( ) 0 58 122 2.1034dfdα α αα

= = − => =

Gradient Search Method: Box Evolutionary 2 it is a geometrical figure in four or more dimensions which is analogous to a cube in three dimensions.

In geometry, a hypercube is an n-dimensional analogue of a square (n = 2) and a cube (n = 3). It is a closed, compact, convex figure whose 1-skeleton consists of groups of opposite parallel line segments aligned in each of the space's dimensions, perpendicular to each other and of the same length. A unit hypercube's longest diagonal in n-dimensions is equal to √n.

2 2i i∆ = ∆ + ∆

Box Evolutionary : Example 1


Corresponding point is

2 22 2 2.828∆ = + =

Next fig shows


2 22 2 2.828∆ = + =


2 21 1 1.414∆ = + =

( )0x x=


2 21 1 1.414∆ = + =

Simplex Search Method In the simplex search method the number of points in the initial simplex is much less compared to Box Evolutionary method. With N variable (N+1) number of variables are used in the initial simplex. In each iteration worst point (xh) in the simplex is found first. Then a new simplex with xnew is formed. Four different situation may arise depend on function value.

First, the centroid (xc) of all but worst point is determined.

Thereafter, the worst point in the simplex is reflected about the xc and a new point xr is found.

if

Simplex Search Method Usually γ ≈ 2.0, β = 0.5

If function value at reflected point is worse than the worst point in the simplex, a contraction is made with β is negative

If function value at reflected point is better than the worst and worse than the next to worst point in the simplex, a contraction is made with β is positive

centroid (xc) of all but worst point .

f (xg) f (xh) f (xl)

f (x)

β +ve β -ve γ +ve

Even though the reflected point (xr) is better than the new point (xnew), the basic simplex search algorithm does not allow this point in the new simplex.

Simplex Search Method : Example 1

Simplex Search Method : Example 1 1 221

1

( ( ) ( ))1

Ni c

i

f x f xQN

+

=

−= + ∑

1 2(1) 2

(2) 2 (3) 2

( ( ) ( ))3

( ( ) ( )) ( ( ) ( ))3 3

c

c c

f x f x

f x f x f x f x

− =

− − + +

f(x(1))=21.4 ; f(x(2))=74; f(x(3))=106, f(xc)=95.6 =>Q=45.01

18.395

40.40

Gradient Based Method: Cauchy’s steepest method 1

2

f xf

f x∂ ∂

∇ = ∂ ∂ Step 1:

Step 2:

Step 3:

Step 4:

Step 5:

1 1( )f f X∇ =∇

11

2

00

xX

x

= =

1 1( )S f X= −∇

To find X2, optimal step length α1* is required to be find out, therefore minimize f(X1+α1S1) with respect to α1. As df1/dα1=0, α1=α1* and we get

*2 1 1 1X X Sα= +

If , Terminate or proceed to further iteration to get X3 and corresponding value of In new iteration ........

2 2( ) 0f f X∇ =∇ =

3f∇

2 2( )S f X= −∇

To find X2, optimal step length α2* is required to be find out, therefore minimize f(X2+α2S2) with respect to α2. As df2/dα2=0, α2=α2* and we get

*3 2 2 2X X Sα= +

Convergence limit:

11

( ) ( )( )

i i

i

f X f Xf X

ε+ −≤ 2

i

fx

ε∂≤

∂ 1 3i iX X ε+ − ≤

Gradient Based Method: Cauchy’s steepest method

1 1

1( )

1S f X

− = −∇ =

Gradient Based Method: Cauchy’s steepest method

Alternative steps for Cauchy’s Steepest descent

Cauchy’s method works well when x(0)

is far away from x*. When the current

point is very close to the minimum, the change in the gradient vector is small. Thus the new point

created by the unidirectional search is also close to the current point.

( ) ( )( )( )

( ) ( ) ( ) ( ) ( )( ) 2t

t t t t ti i i i i

i x

f x f x x f x x xx

∂= + ∆ − −∆ ∆

∂

( )

( )

1 2

( ) , ......t

Tt

Nx

f f ff xx x x

∂ ∂ ∂∇ = ∂ ∂ ∂

Analytically Numerically by central difference scheme

Cauchy’s Steepest descent: Example 1 ∂f/∂x1 = 2*x1 + 4*x1*(x1^2 + x2 - 11) + 2*x2^2 - 14 ∂f/∂x2= 2*x2 + 4*x2*(x2^2 + x1 - 7) + 2*x1^2 - 22

( )

( )

( ) t

tx x

sα α−

=

(1.788,2.840) (0,0) 0.127(2,5)

T T

T α−= =

Slope 14:22

3,2

1.788, 2.810

By geometry

Cauchy’s Steepest descent: Example 1

For golden search method please see next slide ............

SVO: Golden Search Method

SVO: Golden Search Method : Example 1

functions of a single variable - indian institute of ...shibayan/mcc 52105... · gradient of a...

Documents