classical optimization theory unconstrained problem

Classical Optimization Theory

B S V P Surya TejaK Rohit

B Surya TejMrudul Nekkanti

Find Out the type of extreme points in the following figures.

Now what about this?

• This one has no minima or maxima.

• The minima or maxima are defined in a specific region which, in other words, we call as constrained problems.

What is minimum or maximum?

▪ Minimum, in mathematical terms, for a function can be defined as

▪ f (X0 + h) > f (X0) V X0 ∈ domain

▪ Similarly, maximum can be defined, for a function, as the following

▪ f (X0 + h) < f (X0) V X0∈ domain

▪ Here “h” is a small value and tends to zero.

▪ These are local maxima and minima, because we are not basing them for the whole domain.

▪ But if we take the smallest value of all the local minimas, then the value is called a Global Minima.

▪ If we take the largest of all the local maximas, then it is called as Global Maxima.

Necessary Conditions

▪ We are going to develop necessary and sufficient conditions for an n-variable function f(X) to have extrema.

▪ It is assumed that the first and second order partial derivatives of f(X) are continuous for all X.

▪ Theorem 1: A necessary condition for X0 to be an extreme point of f(X) is that ∇ f(X0) = 0

Sufficient conditions

▪ Theorem: A sufficient condition for a stationary point X0 to be an extreme point is that the Hessian matrix H evaluated at X0 satisfy the following conditionsi. H is positive definite, if X0 is a minimum point

ii. H is negative definite, if X0 is a maximum point

Hessian Matrix

The Newton Raphson Method

▪ The necessary condition, sometimes ∇ f(X) = 0 , can be difficult to solve numerically.

▪ So we use an iterative method called Newton Raphson method, which helps solving simultaneous nonlinear equations.

▪ The method is mentioned in the next slides.

▪ Consider the simultaneous equation f i(X) = 0, i = 1,2,3 … m

▪ By Taylor’s expression at a given point Xk , we can write the whole expression in the following form

f i(X) ~ f i(Xk) + ∇ fi(Xk)(X - Xk)Changing the equation will give us the following expression

f i(Xk) + ∇ fi(Xk)(X - Xk) = 0

This can be written as Ak +Bk(X - Xk) = 0

OR X = Xk - B-1k Ak (Bkis non

singular)

▪ The whole idea of this method is to start from an initial point and then move on by using the above equation to find a point until it converges.

▪ This process is done until 2 successive points are almost equal.

▪ For a single variable function this can be shown as

xk + 1 = xk -

Example

Demonstrate Newton Raphson Method on the following

g(x) = (3x - 2) 2(2x - 3)2

First find out f(x) = g’(x) = 72x3 - 234x2 + 241x – 78

Then follow the newton Raphson equation for a single variable that is shown below.

xk + 1 = xk -

Solving

k xk f(xk)/f'(xk) xk+1

0 10 2.967892314 7.032107686

1 7.032107686 1.97642875 5.055678936

2 5.055678936 1.314367243 3.741311693

3 3.741311693 0.871358025 2.869953668

4 2.869953668 0.573547408 2.29640626

5 2.29640626 0.371251989 1.925154272

6 1.925154272 0.230702166 1.694452106

7 1.694452106 0.128999578 1.565452528

8 1.565452528 0.054156405 1.511296123

9 1.511296123 0.010864068 1.500432055

10 1.500432055 0.000431385 1.50000067

11 1.50000067 6.70394E-07 1.5

▪ It converges at 1.5

▪ Taking some other initial value we can converge at the other points. Initial values 1 and 0.5 should give the other 2 extreme points.

questions

Thank You!