The Most Important Concept in Optimization (minimization)
A point is said to be an optimal solution of a unconstrained minimization if there exists no decent direction
A point is said to be an optimal solution of a constrained minimization if there exists no feasible decent direction
There might exist decent direction but move along this direction will leave out the feasible region
Decent Direction of
Move alone the decent direction for a certain stepsize will decrease the objective function value i.e.,
f (x0+ õd) < f (x0); õ 2 (0; î )
d 2 Rn is descent direction if9 î > 0; such that
r f (x0)d < 0
x0
Feasible Direction of
Move alone the feasible direction from for a certain stepsize will not leave the feasible region i.e.,
x0
x0
(x0+ õd) 2 F; õ 2 (0; î )
d 2 Rn is f easible direction of x0 2 F
if 9 î > 0; such that
Fwhere is the feasible region.
minx2R2
(x1à 3)2+ x22
minx2R3
(x1à 2)4+ x21x
22+ (x2à 1)2+ (x1+ x3)2
Steep Decent with Exact Line Search
Start with any
x0 2 Rn . Having xi 2 Rn
stop if r f (xi) = 0(i) Steep Decent Direction :
di = à r f(xi)0
(ii) Finding Stepsize by Exact Line Search :
õã 2 argminõ>0
f (xi + õdi)
xi+1 = xi + õãdi
Minimum Principle
Let f : Rn ! Rbe a convex and differentiable function
F ò Rnbe the feasible region.
xã 2 argminx2F
f (x) ( ) r f (xã)(x à xã) > 0 8x 2 F
Example:min (x à 1)2 s:t: a ô x ô b
minx2R2
x21+ x2
2
x1+ x264à x1à x2 6 à 2
x1; x2>0
ï
ï
ï ï
Minimization Problem vs.
Kuhn-Tucker Stationary-point Problem
r f (x) + ë0r g(x) = 0
g(x)60;
ë0g(x) = 0
ë > 0
Find x 2 Ò; ë 2 Rmsuch thatKTSP:
minx2Ò
f (x)MP: such that
g(x)60
Lagrangian Function
For a fixed
L(x;ë) = f (x) + ë0g(x)
Let L(x;ë) = f (x) + ë0g(x) and ë > 0 If f (x);g(x)are convex thenL (x;ë)is convex.
ë > 0, if x 2 argminf L (x;ë)j x 2 Rngthen
@x@L(x;ë)
ìììx=x
= r f (x) +ë0r g(x) = 0
Above result is a sufficient condition ifL (x;ë)
is convex.
KTSP with Equality Constraints?
(Assumeh(x) = 0are linear functions)
h(x) = 0 , h(x)60 and à h(x)60
KTSP:
r f (x) + ë0r g(x) + (ì + à ì à )0r h
g(x)60;
ë0g(x) = 0;
ë>;
Find x 2 Ò; ë 2 Rk; ì +; ì à 2 Rmsuch that
(x) = 0
(ì +)0h(x) = 0; (ì à )0(à h(x)) = 0
h(x) = 0
ì +; ì à > 0
KTSP with Equality Constraints
KTSP: Find x 2 Ò; ë 2 Rk; ì 2 Rm such that
r f (x) + ë0r g(x) + ì r h
g(x)60;ë0g(x) = 0;
ë>0
(x) = 0
h(x) = 0
If ì = ì + à ì à and ì +; ì à > 0 then
ì is free variable
Generalized Lagrangian FunctionL(x;ë; ì ) = f (x) + ë0g(x) + ì 0h(x)
For fixed ë>0;ì , if x 2 argminf L (x;ë;ì )j x 2 Rngthen
Let and ë > 0L(x;ë; ì ) = f (x) + ë0g(x) + ì 0h(x)
L (x;ë;ì )
If f (x);g(x)are convex and is linear thenh(x)is convex.
@x@L(x;ë;ì )
ìììx=x
= r f (x) +ë0r g(x) + ì 0r h(x) = 0
Above result is a sufficient condition if
is convex.
L (x;ë;ì )
Lagrangian Dual Problem
maxë;ì
minx2Ò
L(x;ë; ì )
subject to ë > 0
Lagrangian Dual Problem
maxë;ì
minx2Ò
L(x;ë; ì )
subject to ë > 0
maxë;ì
ò(ë; ì )
subject to ë > 0where ò(ë; ì ) = inf
x2ÒL(x;ë; ì )
Weak Duality Theorem
Let x 2 Òbe a feasible solution of the primal
problem and(ë; ì )a feasible solution of the
dual problem. Then f (x)>ò(ë; ì )
Corollary: supfò(ë; ì )j ë>0g
6 inff f (x)j g(x) 6 0; h(x) = 0g
ò(ë; ì ) = infx2Ò
L(x;ë; ì ) ô L (xà;ë; ì )
Weak Duality Theorem
06ë ? g(x)60
Corollary: ëã>0If f (xã) = ò(ëã; ì ã)where
and g(xã)60;h(xã) = 0 , then xã and(ëã; ì ã)
solve the primal and dual problem respectively.In this case,
Saddle Point of Lagrangian
Let xã 2 Ò;ëã>0; ì ã 2 Rmsatisfying
L (xã;ë; ì )6L (xã;ëã; ì ã) 6L(x;ëã; ì ã);
8 x 2 Ò; ë>0: Then (xã;ëã; ì ã) is called
The saddle point of the Lagrangian function
Dual Problem of Linear Program
minx2R n
p0x
subject to Ax > b; x>0
Primal LP
Dual LP maxë2R m
b0ë
subject to A0ë6p; ë>0
※ All duality theorems hold and work perfectly!
Application of LP Duality (I) Farkas’ Lemma
For any matrixA 2 Rmâ nand any vectorb2 Rm;either
Ax60; b0x > 0 has a solution
or
A0ë = b; ë>0 has a solution
but never both.
Application of LP Duality (II) LSQ-Normal Equation Always Has a Solution
For any matrixA 2 Rmâ nand any vectorb2 Rm;
consider minx2R n
jjAx à bjj22
xã 2 argminf jjAx à bjj22g , A0Axã = A0b
Claim: A0Ax = A0balways has a solution.
Dual Problem of Strictly Convex Quadratic Program
minx2R n
21x0Qx + p0x
subject to Ax6 b
Primal QP
With strictly convex assumption, we have
Dual QP
max à 21(p0+ ë0A)Qà 1(A0ë + p) à ë0b
subject to ë>0
