the most important concept in optimization (minimization)

The Most Important Concept in Optimization (minimization)

A point is said to be an optimal solution of a unconstrained minimization if there exists no decent direction

A point is said to be an optimal solution of a constrained minimization if there exists no feasible decent direction

There might exist decent direction but move along this direction will leave out the feasible region

Decent Direction of

Move alone the decent direction for a certain stepsize will decrease the objective function value i.e.,

f (x0+ õd) < f (x0); õ 2 (0; î )

d 2 Rn is descent direction if9 î > 0; such that

r f (x0)d < 0

x0

Feasible Direction of

Move alone the feasible direction from for a certain stepsize will not leave the feasible region i.e.,

x0

x0

(x0+ õd) 2 F; õ 2 (0; î )

d 2 Rn is f easible direction of x0 2 F

if 9 î > 0; such that

Fwhere is the feasible region.

minx2R2

(x1à 3)2+ x22

minx2R3

(x1à 2)4+ x21x

22+ (x2à 1)2+ (x1+ x3)2

Steep Decent with Exact Line Search

Start with any

x0 2 Rn . Having xi 2 Rn

stop if r f (xi) = 0(i) Steep Decent Direction :

di = à r f(xi)0

(ii) Finding Stepsize by Exact Line Search :

õã 2 argminõ>0

f (xi + õdi)

xi+1 = xi + õãdi

Minimum Principle

Let f : Rn ! Rbe a convex and differentiable function

F ò Rnbe the feasible region.

xã 2 argminx2F

f (x) ( ) r f (xã)(x à xã) > 0 8x 2 F

Example:min (x à 1)2 s:t: a ô x ô b

minx2R2

x21+ x2

2

x1+ x264à x1à x2 6 à 2

x1; x2>0

ï

ï

ï ï

Minimization Problem vs.

Kuhn-Tucker Stationary-point Problem

r f (x) + ë0r g(x) = 0

g(x)60;

ë0g(x) = 0

ë > 0

Find x 2 Ò; ë 2 Rmsuch thatKTSP:

minx2Ò

f (x)MP: such that

g(x)60

Lagrangian Function

For a fixed

L(x;ë) = f (x) + ë0g(x)

Let L(x;ë) = f (x) + ë0g(x) and ë > 0 If f (x);g(x)are convex thenL (x;ë)is convex.

ë > 0, if x 2 argminf L (x;ë)j x 2 Rngthen

@x@L(x;ë)

ìììx=x

= r f (x) +ë0r g(x) = 0

Above result is a sufficient condition ifL (x;ë)

is convex.

KTSP with Equality Constraints?

(Assumeh(x) = 0are linear functions)

h(x) = 0 , h(x)60 and à h(x)60

KTSP:

r f (x) + ë0r g(x) + (ì + à ì à )0r h

g(x)60;

ë0g(x) = 0;

ë>;

Find x 2 Ò; ë 2 Rk; ì +; ì à 2 Rmsuch that

(x) = 0

(ì +)0h(x) = 0; (ì à )0(à h(x)) = 0

h(x) = 0

ì +; ì à > 0

KTSP with Equality Constraints

KTSP: Find x 2 Ò; ë 2 Rk; ì 2 Rm such that

r f (x) + ë0r g(x) + ì r h

g(x)60;ë0g(x) = 0;

ë>0

(x) = 0

h(x) = 0

If ì = ì + à ì à and ì +; ì à > 0 then

ì is free variable

Generalized Lagrangian FunctionL(x;ë; ì ) = f (x) + ë0g(x) + ì 0h(x)

For fixed ë>0;ì , if x 2 argminf L (x;ë;ì )j x 2 Rngthen

Let and ë > 0L(x;ë; ì ) = f (x) + ë0g(x) + ì 0h(x)

L (x;ë;ì )

If f (x);g(x)are convex and is linear thenh(x)is convex.

@x@L(x;ë;ì )

ìììx=x

= r f (x) +ë0r g(x) + ì 0r h(x) = 0

Above result is a sufficient condition if

is convex.

L (x;ë;ì )

Lagrangian Dual Problem

maxë;ì

minx2Ò

L(x;ë; ì )

subject to ë > 0

Lagrangian Dual Problem

maxë;ì

minx2Ò

L(x;ë; ì )

subject to ë > 0

maxë;ì

ò(ë; ì )

subject to ë > 0where ò(ë; ì ) = inf

x2ÒL(x;ë; ì )

Weak Duality Theorem

Let x 2 Òbe a feasible solution of the primal

problem and(ë; ì )a feasible solution of the

dual problem. Then f (x)>ò(ë; ì )

Corollary: supfò(ë; ì )j ë>0g

6 inff f (x)j g(x) 6 0; h(x) = 0g

ò(ë; ì ) = infx2Ò

L(x;ë; ì ) ô L (xà;ë; ì )

Weak Duality Theorem

06ë ? g(x)60

Corollary: ëã>0If f (xã) = ò(ëã; ì ã)where

and g(xã)60;h(xã) = 0 , then xã and(ëã; ì ã)

solve the primal and dual problem respectively.In this case,

Saddle Point of Lagrangian

Let xã 2 Ò;ëã>0; ì ã 2 Rmsatisfying

L (xã;ë; ì )6L (xã;ëã; ì ã) 6L(x;ëã; ì ã);

8 x 2 Ò; ë>0: Then (xã;ëã; ì ã) is called

The saddle point of the Lagrangian function

Dual Problem of Linear Program

minx2R n

p0x

subject to Ax > b; x>0

Primal LP

Dual LP maxë2R m

b0ë

subject to A0ë6p; ë>0

※ All duality theorems hold and work perfectly!

Application of LP Duality (I) Farkas’ Lemma

For any matrixA 2 Rmâ nand any vectorb2 Rm;either

Ax60; b0x > 0 has a solution

or

A0ë = b; ë>0 has a solution

but never both.

Application of LP Duality (II) LSQ-Normal Equation Always Has a Solution

For any matrixA 2 Rmâ nand any vectorb2 Rm;

consider minx2R n

jjAx à bjj22

xã 2 argminf jjAx à bjj22g , A0Axã = A0b

Claim: A0Ax = A0balways has a solution.

Dual Problem of Strictly Convex Quadratic Program

minx2R n

21x0Qx + p0x

subject to Ax6 b

Primal QP

With strictly convex assumption, we have

Dual QP

max à 21(p0+ ë0A)Qà 1(A0ë + p) à ë0b

subject to ë>0