Part 4 Nonlinear Programming

4.1 Introduction

Standard Form

min

. .

0 1, 2, ,

0 1, 2, ,

j

j

f

s t

h j m

g j m m p

x

x

x

An Intuitive Approach to Handle the Equality Constraints

One method of handling just one or two equality constraints is to solve for 1 or 2 variables and eliminate them from problem formulation by substitution.

2 21 2 1 2

22 2 2 21 2 1 1 1 1 1

1 1 21

EX. s.t. 1

Soln:

1 2 2 1 ( )

1 14 2 0

2 2

f x x x x

f x x x x x x f x

dfx x x

dx

x

x

Use of Lagrange Multipliers to Handle m Equality Constraints and m+n Variables

1 2

1 1 2

2 1 2

1 2

min , , ,

. .

, , , 0

, , , 0

, , , 0

n m

n m

n m

n n m

f y y y

s t

h y y y

h y y y

h y y y

Equivalent Formulation

1 1

1 1 1

1 1

state variables decision variables

min , , ; , ,

. .

, , ; , , 0

, , ; , , 0

n m

n m

n n m

f x x u u

s t

h x x u u

h x x u u

min ; (1)

. .

( ; ) (2)

f

s t

x u

h x u 0

Choice of Decision Variables

For a given optimization problem, the choice of which variables to designate as the decision variables is not unique.

It is only a matter of convenience to make a distinction between decision and state variables.

First Derivation of Necessary Conditions (i)

A stationary point is one where

0 (3)

for arbitrary while holding

(4)

and letting change as it will.

f fdf d d

d

d d d

d

x ux u

u

h hh 0 x u

x u

x

1 2

1

2

1 1 1

1 2

2 2 2

1 2

1 2

n

n

n

n

n n n

n

f f f f

x x x

dx

dxd

dx

h h h

x x x

h h h

x x x

h h h

x x x

x

x

h

x

First Derivation of Necessary Conditions (ii)

n n

1

If is nonsingular (and it should be if determines from Eq (2)),

Eq (4) can be solved for , i.e.

d

d

d

d d

x u

hu x

x

x

x h h u

1

(5)

Substituting into Eq (3) yields

0 (6)

Hence, if is to be zero for arbitrary , it is necessary that

df f f d

df d

f f

u x x u

u x x

h h u

u

h

1 ( equations) (7)

These equations together with

; ( equations)

determine and .

m

m

n

uh 0

h x u 0

u x

First Derivation of Necessary Conditions (iii)

In other words, Eq (7) represents

But, notice that, in general

f

f f

h

x

u

u u

Second Derivation of Necessary Conditions (i)

* * *

Consider first a special case

min , ,

. .

, , 0

, , 0

At extremum ( , , )

0

Since , and are not independent, we cannot conclude

that , and vanish identically.

x y z

x y z

f x y z

s t

g x y z

h x y z

x y z

df f dx f dy f dz

dx dy dz

f f f

Second Derivation of Necessary Conditions (ii)

1 2

1 2

1 2

1 2

Since and must be maintained constant at zero,

0

0

Let us introduce two constants and ,

0

x y z

x y z

x x x

y y y

z z z

g h

dg g dx g dy g dz

dh h dx h dy h dz

f g h dx

f g h dy

f g h dz

Second Derivation of Necessary Conditions (iii)

1 2

1 2

1 2

Now, it is possible to find and such that at least

one pair of the differentials are zero, say

0 (8)

0 x x x

y y y

f g h

f g h

(9)

Otherwise,

0

and and would be functionally dependent and, thus,

they must be either equivalent or inconsistent.

x x z zy y

y y x xz z

g h g hg h

g h g hg h

g h

Second Derivation of Necessary Conditions (iv)

1 2

If and are determined by Eqs (8) and (9), the remaining

differentials can be arbitrarily assigned and forced to zero.

0 (10)

Eqs (8), z z zf g h

(9) and (10) together with

, , 0

, , 0

can be solved simultaneously.

g x y z

h x y z

Second Derivation of Necessary Conditions - General Formulation

min ;

. . ;

Necessary Conditions are:

( equations) (11)

( equations) (12)

Together with

; ( equations)

From Eq (11),

T

T

T

f

s t

f n

f m

n

f

x x

u u

x u

h x u 0

λ h 0

λ h 0

h x u 0

λ 1

1

Substituting this into Eq (12) yields Eq (7)

f f

x x

u x x u

h

h h 0

Derivation with Lagrange Multipliers

1 2

1 2

Adjoin the constraints to the objective function by

a set of undetermined multipliers, , , , , i.e.

, , , ,

where , , , are called Lagrange multipliers.

Treat the minimization proble

n

T

n

n

L f

x u λ x u λ h x u

, ,

m

min , ,

as an unconstrained problem. The necessary conditions are:

which are the same as before.

T

T

L

L f

L f

L

x u λx u λ

hλ 0

x x xh

λ 0u u u

h 0λ

Example:

Solution:

2 2

2 2

1 subject to , 0

2

x uf h x u x mu c

a b

2 2

2 2

2 2

2 2

2 2 2 2 2 2 2 2 2

1

2

0; 0; 0

; ;

x uL f h x mu c

a b

L x L u Lm x mu c

x a u b

mcb ca cu x

a m b a m b a m b

1 2 1 2

2 21 2 1 2

2 21 2 1 2

11

22

* *1 2

* *

min ,

. . , 0 and 1

1

1 2 0

1 2 0

10.707;

1.4141 1.414

;1 1.414

f y y y y

s t h y y y y b b

L y y y y

Ly

y

Ly

y

y y

f h

Example :

Solution :

y y

Sensitivity Interpretation

1/ 2* *1 2

1/ 2*

1/ 2* *1 2

1/ 2 *

*

1

2

2

2

2

1 1 1 (1) 1b

by b y b

b b

V b y b y b b

dVb b

dbdV

V b V b V bdb

Generalized Sensitivity

* *

*

min

. .

ˆ 1, 2, ,

If and the optimal objective value is .

The corresponding local optimum is and .

The constraint with the largest absolute value is the

one whos

i i

ii

i

f

s t

h b i m

V

V

b

b

y

y

b = b b

y b λ b

b

e rhs affects the optimal value function the most,

at least for close to .

V

b b

Problems with Inequality Constraints Only

min

. .

f

s t

y

g y 0

One Constraint and One Variable

*

* *

* *

*

* *

*

*

Two possibilities at minimum: (1) ( ) 0 and (2) ( ) 0.

(1) If ( ) 0, the constraint is not effective and can be ignored.

0

(2) If ( ) 0, then 0 and 0

sgn

y

y yy y

y

g y g y

g y

df

dy

df dgg y df dy dg dy

dy dy

df

dy

*

sgn

The above two possibilities can be expressed in one equation as

0 and 0 (13)

y

dg

dy

df dg

dy dy

( )f y ( )f y

( )f y

( )f y ( )f y

Two Possibilities at Minimum

* *( ) 0g y a y

* *( ) 0g y a y

*

0

0

y

df

dy

* *

* *

0 and 0

or sgn sgn

y y

y y

df dg

dy dy

df dg

dy dy

**y *y a

One Constraint and Two Variables

Eq (13) can be written as

(14)

which should be interpreted as: parallel to

but pointing in opposite directions.

f g

f g

0

f

g

f

0g

constantf

Area of improvement

J Inequality Constraints and N Variables

1 1 2 2

1 2

1 2

The necessary condition is:

or

0 , , , 0where,

0 , , , 0

and 1, 2, ,

Since, at minimum, 's have to be nonnegative, can be

expressed as a

T

J J

i Ni

i N

i

f

f g g g

g y y y

g y y y

i J

f

μ g 0

0

negative linear combination of 's. In words,

the gradient of w.r.t at a minimum must be pointed in such

a way the decrease of can only come by violating the constraints.

jg

f

f

y

2-D Case

constantf 1 0g

2 0g

1 0g

2 0g

constantf

f

1g2g

Kuhn-Tucker Conditions: Geometrical Interpretation

At any local constrained optimum, no (small) allowable change in the problem variables can improve the value of the objective function.

lies within the cone generated by the negative gradients of the active constraints.

f

General Formulation

1 2

1 2

1 2

min , , , (1)

s.t.

, , , 0 1, 2, , (2)

, , , 0 1, 2, , (3)

N

j N

k N

f y y y

g y y y j J

h y y y k K

Active Constraints

The inequality constraint 0 is said to be an

active or binding constraint at the point if

=0

It is said to be inactive or nonbinding if

0.

j

j

j

g

g

g

y

y

y

y

Kuhn-Tucker Conditions

(4)

(5)

(6)

T T Tf

y y yμ g y λ h y 0

g y 0

h y 0

μ 0

(7)

(8)

or 0 1, 2, ,

T

j jg j J

μ g y 0

y

Kuhn-Tucker Necessity Theorem

*

* *

Consider the NLP problem given by Eqs (1) - (3). Let , and

be differentiable functions and be a feasible solution to NLP.

Let | 0 . Furthermore, for and

for 1,2, , are

j j k

f

j g g j h

k K

y y

g h

y

I y I y

*

* * * * *

linearly independent. If is an optimal solution

to NLP, then there exists a such that solves

the Kuhn-Tucker problem given by Eqs (4) - (8).

y

μ λ y μ λ

21 2

2 21 2 1 1 2

Ex. min

. . 6, 1 0, 26

f y y

s t y y y y y

y

1

1 2 1 2

1 1 2 1 1

1 2 2 1

1 1

2 22 1 2

Solution:

2 1

1 0 , 2 2

1 1

Eq (4) becomes

2 1 2 1 0

1 0 2 1 0

Eq (8) becomes

1 0

26 0

f y

g g y y

h

y y

y

y

y y

1 2

2 21 2

2 21 2 1 2

2 11

1 22

2 21 2

min

. . 25

, 25

2 0

2 0

25 0

f y y

s t g y y

L y y y y

Ly y

y

Ly y

y

y y

Example

y

y

y

Sensitivity

1 2

1 2 1 2

1 2 1 2

1 1

*

*

min , , ,

s.t.

ˆ, , , , , , 0 1, 2, ,

ˆ, , , , , , 0 1, 2, ,

, ,

N

j N j N j

k N k N k

K J

k k j jk j

kk

jj

f y y y

g y y y g y y y c j J

h y y y h y y y b k K

L f h g

V

b

V

c

y λ μ y y y

Constraint Qualification

• When the constraint qualification is not met at the optimum, there may or may not exist a solution to the Kuhn-Tucker problem.

• The Kuhn-Tucker necessity theorem helps to identify points that are not optimal. On the other hand, if the KTC are satisfied, there is no assurance that the solution is truly optimal.

* for and for 1,2, , are linear

independent at the optimum.

j kg j h k K y I

Second-Order Optimality Conditions

2

*

*

, , 0

for all nonzero vectors such that

where is the matrix whose rows are the gradients

of the constraints that are active at . In other words,

the above equation defines a set

T L

yz y λ μ z

z

J y z 0

y

of vectors y that are

orthogonal to the gradients of the active constraints.

These vectors form the to

the active constraints.

tangent plane

Necessary and Sufficient Conditions for Optimality

If a Kuhn-Tucker point satisfies the second-order sufficient conditions, then optimality is guaranteed.

Basic Idea of Penalty Methods

min

. ., , ,

where , , , is a penalty function,

and is a penalty parameter.

f

s tP f r

P f r

r

x

g hg x 0

h x 0

g h

Example

2 2

1 2

1 2

2 2 2

1 2 1 2

min 1 2

. . 4 0

, 1 2 4

f x x

s t h x x

P r x x r x x

x

x

x

Exact L1 Penalty Function

(1) (2) (1) (2)1

1 1

* * *

(1) *

(2) *

, , max 0,

where and are positive.

If , , satisfy the Kuhn-Tucker conditions and if

1, 2, ,

1, 2, ,

then it can be shown tha

K J

k k j jk j

k k

j j

P f w h w g

w k K

w j J

x w w x x x

x λ μ

* (1) (2)1

1

t is a local minimum of , , .

However, is nonsmooth at 0 and 0.k j

P

P h g

x x w w

x x

Equivalent Smooth Constrained Problem

(1) (1) (1) (2) (2)

1 1

(1) (1)

(2) (2)

(1) (1)

(1) (1)

(2)(2) (2)

(1) (1) (2) (2)

min

. .

1, 2, ,

1, 2, ,

0max 0,

0

, , , 0

K J

k k k j jk j

k k k

j j j

k k k

k k

j jj j

k k j j

f w p n w p

s t

h p n k K

g p n j Jp n h

p np g

p n

p n p n

x

x

xx

x

Barrier Method

2 2

1 2

1 2

2 2

1 2 1 2

min 1 2

. . g 4 0

, 1 2 ln 4

where is a positive scalar called the barrier parameter.

f x x

s t x x

B r x x r x x

r

x

x

x

Generalized Case

1

min

. . 0 1, 2, ,

min , ln

Barrier method is not directly applicable to problems

with equality constraints, but equality constraints can be

integrated using a penalty term

j

J

jj

f

s t g j J

B r f r g

x

x

x x x

and inequality can use a

barrier term, leading to a mixed penalty-barrier method.