mit and james orlin © 2003 1 nonlinear programming theory

MIT and James Orlin © 2003

1

Nonlinear Programming Theory


2

Difficulties of NLP Models

Nonlinear Programs:

Linear Program:


3

Graphical Analysis of Non-linear programs

in two dimensions: An example

2 214 15( ) ( )x y Minimize

subject to (x - 8)2 + (y - 9)2 49 x 2 x 13 x + y 24


4

Where is the optimal solution?

0 2 4 6 8 10 12 14 16 18

0

2

4

6

8

10

12

14

16

18y

x

Note: the optimal solution is not at a corner point. It is where the isocontour first hits the feasible region.


5

Another example:

0 2 4 6 8 10 12 14 16 18

0

2

4

6

8

10

12

14

16

18y

x

Minimize

(x-8)2 + (y-8)2

Then the global unconstrained minimum is also feasible.

The optimal solution is not on the boundary of the feasible region.


6

Local vs. Global Optima

There may be several locally optimal solutions.x

z

10

z = f(x)max f(x)s.t. 0 x 1

A

B

C

Def’n: Let x be a feasible solution, then

– x is a global max if f(x) f(y) for every feasible y.

– x is a local max if f(x) f(y) for every feasible y sufficiently close to x (i.e., xj-yj xj+ for all j and some small ).


7

When is a locally optimal solution also globally optimal?

For minimization problems

– The objective function is convex.

– The feasible region is convex.

W

P

2 4 6 8 10 12 14

24

68

1012

14

Convexity and Extreme PointsWe say that a set S is convex, if for every two points x and y in S, and for every real number in [0,1], x + (1-)y S.

The feasible region of a linear program is convex.

x

y

We say that an element w S is an extreme point (vertex, corner point), if w is not the midpoint of any line segment contained in S.

8


9

On convex feasible regions

If all constraints are linear, then the feasible region is convex


10

On Convex Feasible Regions

The intersection of convex regions is convex


11

S

Recognizing convex sets

Rule of thumb: suppose for all x, y S the midpoint of x and y is in S. Then S is convex.

xy

It is convex if the entire line segment is always in S.

(x+y)/2

Which are convex?

CB

B C B C B C

DA

12


13

Joining two points on a curveThe line segment joining two points on a curve. Let f( ) be a function, and let g(y+(1-z)) = f(y) + (1-)f(z) for 0.

f(y)

f(z)

(y+z)/2

f(y)/2 +f(z)/2

g(x)

g(y) = f(y)

g(z) = f(z)

g(y/2 + z/2) = f(y)/2 + g(z)/2

y z

f(x)

x


14

Convex FunctionsConvex Functions: f( y + (1-)z) f(y) + (1-)f(z) for every y and z and for 0.

e.g., = 1/2 f(y/2 + z/2) f(y)/2 + f(z)/2

Line joining any points is above the curvef(x)

xy z

f(y)

f(z)

(y+z)/2

f(y)/2 +f(z)/2

We say “strict” convexity if sign is “<” for 0.


15

Concave FunctionsConcave Functions: f( y + (1-)z) f(y) + (1-)f(z) for every y and z and for 0 .

e.g., = 1/2 f(y/2 + z/2) f(y)/2 + f(z)/2

Line joining any points is below the curve

xy z

f(y)

f(z)

(y+z)/2

f(y)/2 +f(z)/2

We say “strict” concavity if sign is “<” for 0.


16

Classify as convex or concave or both or neither.


17

More on convex functions

x

f(x)

-x

f(-x)

If f(x) is convex, then f(-x) is convex.


18


x

y

If f(x) is convex, then K - f(x) is concave.

x

y


19


If f(x) is a twice differentiable function of one variable, and if f”(x) > 0 for all x, then f(x) is convex.

f(x) = x2.

f’(x) = 2x, f”(x) = 2

0

1

23

4

5

6

78

9

10

-4 -3 -2 -1 0 1 2 3 4

f(x) = - ln(x) for x > 0

f’(x) = -1/x, f”(x) = 1/x2

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

0 0.5 1 1.5 2 2.5 3 3.5 4


20

0

1

2

3

4

5

6

7

8

-3 -2 -1 0 1 2 3f(x) g(x)

Even more on convex functions

If f(x) is convex and g(x) is convex, then so is f(x) + g(x)

f(x)+g(x)


21

0

1

2

3

4

5

6

7

8

-3 -2 -1 0 1 2 3f(x) g(x)

Even more on convex functions

If f(x) is convex and g(x) is convex, then so is max [f(x), g(x)]

max[f(x), g(x)]


22

What functions are convex?

f(x) = 4x + 7 all linear functions

f(x) = 4x2 – 13 some quadratic functions

f(x) = ex

f(x) = 1/x for x > 0

f(x) = |x|

f(x) = - ln(x) for x > 0


23

Convex functions vs. convex sets

If y = f(x) is convex, then {(x,y) : f(x) y} is a convex set

y

x

f(x)


24

Local Minimum Property

A local min of a convex function on a convex feasible region is also a global min.

Strict convexity implies that the global minimum is unique.

The following NLPs can be solved

– Minimization Problems with a convex objective function and linear constraints


25

Local minimum property

There is a unique local minimum for the function below.

y

x

f(x)

The local minimum is a global minimum


26

Local Maximum Property

A local max of a concave function on a convex feasible region is also a global max.

Strict concavity implies that the global optimum is unique.

Given this, the following NLPs can be solved

– Maximization Problems with a concave objective function and linear constraints


27

Local maximum property

y

x

There is a unique local maximum for the function below.

The local maximum is a global minimum


28

More on local optimality

The techniques for non-linear optimization minimization usually find local optima.

This is useful when a locally optimal solution is a globally optimal solution

It is not so useful in many situations.

Conclusion: if you solve an NLP, try to find out how good the local optimal solutions are.


29

Finding a local optimal for a single variable NLPSolving NLP's with One Variable:

max f()

s.t. a b

Optimal solution is either

a boundary point or

satisfies f() = 0 and f () < 0.

f()

a *

b

f()

a * b

f()

a * b


30

Unimodal Functions

A single variable function f is unimodal if there is at most one local maximum (or at most one local minimum) .


31

Solving Single Variable NLP (contd.)If f() is concave (or simply unimodal) and differentiable

max f()

s.t. a b

Bisection (or Bolzano) Search:

Step 1. Begin with the region of uncertainty for as [a, b]. Evaluate f() at the midpoint =(a+b)/2.

Step 2. If f() > 0, then eliminate the interval up to . If f() < 0, then eliminate the interval beyond .

Step 3. Evaluate f() at the midpoint of the new interval. Return to Step 2 until the interval of uncertainty is sufficiently small.


32

643216 2824

Interval Bisection or Bolzano Search

0

Determine by taking a derivative if a local maximum is to the right or left.

to right

to left


33

-15

-10

-5

0

5

10

15

20

25

30

35

40

0 10 20 30 40 50 60 70


34

Other Search Techniques Instead of taking derivatives (which may be computationally

intensive), use two function evaluations to determine updated interval.

Fibonacci Search

Step 1. Begin with the region of uncertainty for as [a, b]. Evaluate f() and f() for 2 symmetric points <.

Step 2. If f() f(), then eliminate the interval up to . If f()

f(), then eliminate the interval beyond .

Step 3. Select a second point symmetric to the point already in the new interval, rename these points and such that < and evaluate f() and f(). Return to Step 2 until the interval is sufficiently small.


35

On Fibonacci search

1, 1, 2, 3, 5, 8, 13, 21, 34

At iteration 1, the length of the search interval is the kth fibonacci number for some k

At iteration j, the length of the search interval is the k-j+1 fibonacci number.

The technique converges to the optimal when the function is unimodal.


36

Finding a local maximum using Fibonacci Search.

2113 2618

16 190 34

Where the maximum may be

Length of search interval 342113853


37

The search finds a local maximum, but not necessarily a global maximum.

2113 2618

16 190 34


38

The search finds a local maximum, but not necessarily a global maximum.

2113 2618

16 190 34


39

Number of function evaluations in Fibonacci Search

As new point is chosen symmetrically, the length lk of successive search intervals is given by: lk = lk+1 + lk+2 .

Solving for these lengths given a final interval length of 1, ln = 1, gives the Fibonacci numbers: 1, 2, 3, 5, 8, 13, 21, 34,…

Thus, if the initial interval has length 34, it takes 8 function calculations to reduce the interval length to 1.

Remark: if the function is convex or unimodal, then fibonacci search converges to the global maximum


40

NLP Summary Convex and Concave functions as well as convex sets

are important properties

Bolzano and Fibonacci search techniques

– used to solve single variable unimodal functions

mit and james orlin © 2003 1 nonlinear programming theory

Documents

convex slide

points x

y x note

curve fx x y z

x y2 slide

midpoint of x

feasible y

x j y j x j