1

OR IIOR IIGSLM 52800GSLM 52800

2

OutlineOutline classical optimization – unconstrained

optimization dimensions of optimization feasible direction

3

Classical Optimization ResultsClassical Optimization Results Unconstrained Optimization Unconstrained Optimization

different dimensions of optimization conditions nature of conditions

necessary conditions (必要條件 ): satisfied by any minimum (and possibly by some non-minimum points)

sufficient conditions (充分條件 ): if satisfied by a point, implying that the point is a minimum (though some minima may not satisfy the conditions)

order of conditions first-order conditions: in terms of the first derivatives of f & gj

second-order conditions: in terms of the second derivatives of f & gj

general assumptions: f, g, gj C1 (i.e., once continuously differentiable) or C2 (i.e., twice continuously differentiable) as required by the conditions

4

Feasible Direction Feasible Direction S n: the feasible region x S: a feasible point a feasible direction d of x: if there exists >

0 such that x+d S for 0 < <

5

Two Key Concepts Two Key Concepts for Classical Results for Classical Results

f: the direction of steepest accent gradient of f at x0 being orthogonal to

the tangent of the contour f(x) = c at x0

6

The Direction of Steepest Accent The Direction of Steepest Accent ff contours of f(x1, x2) = ff: direction of steepest accent in some sense, increment of unit

move depending on the angle with f f within within 9090 of of ff: increasing: increasing

closer to 0closer to 0: increasing more: increasing more beyond beyond 9090 of of ff: decreasing: decreasing

closer to 180closer to 180: decreasing more: decreasing more above results generally true for above results generally true for

any any ff

x2

x1

2 21 2x x

ff((xx11, , xx22) =) =

ff((xx1010, , xx2020) = ) = cc

d on the tangent plane at xd on the tangent plane at x00

ff(x(x00++d) d) cc for small for small roughly speaking, for roughly speaking, for ff(x(x00) = ) = cc, , ff(x(x00++d) = d) = cc for for

small small when d is on the tangent plane at x when d is on the tangent plane at x00

7

Gradient of Gradient of ff at x at x00 Being Orthogonal to Being Orthogonal to the Tangent of the Contour the Tangent of the Contour ff(x) = (x) = cc at x at x00

2 21 2x x

8

First-Order Necessary Condition First-Order Necessary Condition (FONC)(FONC)

ff CC11 on S and x on S and x** a local minimum of a local minimum of ff then for any feasible direction d at xthen for any feasible direction d at x**, , TTff(x(x**)d )d

0 0 increasing of increasing of ff at any feasible direction at any feasible direction

ff((xx) = ) = xx22 for 2 for 2 xx 5 5 ff((xx, , yy) = ) = xx22 + + yy22 for 0 for 0 xx, , yy 2 2

ff((xx, , yy) = ) = xx22 + + yy22 for for xx 3, 3, yy 3 3

9

FONC for Unconstrained NLPFONC for Unconstrained NLP ff CC11 on S & x on S & x** an interior local minimum an interior local minimum

(i.e., without touching any boundary) (i.e., without touching any boundary) TTff(x(x**) = 0) = 0

10

FONC Not SufficientFONC Not Sufficient Example 3.2.2: f(x, y) = -(x2 + y2) for 0 x, y

Tf((0, 0))d = 0 for all feasible direction d (0, 0): a maximum point

Example 3.2.3: f(x) = x3

f(0) = 0 x = 0 a stationary point

11

Feasible Region with Feasible Region with Non-negativity ConstraintsNon-negativity Constraints

Example 3.2.4. (Example 10.8 of JB) Find candidates of the minimum points by the FONC. min f(x) = subject to x1 0, x2 0, x2 0

* ** *( ) ( )0, if 0; 0, if 0.j j

j j

f fx xx x

x x

2 2 21 2 3 1 2 1 3 13 2 2 2x x x x x x x x

* **( ) ( )0, 0j

j j

f fxx x

x xor, equivalently

12

Second-Order ConditionsSecond-Order Conditions another form of Taylor’s Theorem

f(x) = f(x*)+Tf(x*)(x-x*)

+0.5(x- x*)TH(x*)(x - x*)+ , where being small, dominated by other terms

if Tf(x*)(x-x*) = 0, f(x) f(x*) (x- x*)TH(x*)(x - x*) 0

13

Second-Order Necessary ConditionSecond-Order Necessary Condition

ff CC22 on S on S if xif x** is a local minimum of is a local minimum of ff, then for any , then for any

feasible direction d feasible direction d nn at x at x**, , (i).(i). TTff(x(x**)d )d 0, and 0, and (ii). if (ii). if TTff(x(x**)d = 0, then d)d = 0, then dTTH(xH(x**)d )d 0 0

14

Example 3.3.1(a)Example 3.3.1(a) SONC satisfied

ff((xx) = ) = xx22 for 2 for 2 xx 5 5 ff((xx, , yy) = ) = xx22 + + yy22 for 0 for 0 xx, , yy 2 2

ff((xx, , yy) = ) = xx22 + + yy22 for for xx 3, 3, yy 3 3

15

Example 3.3.1(b)Example 3.3.1(b) SONC: more discriminative than FONCSONC: more discriminative than FONC ff((xx, , yy) = -() = -(xx22 + + yy22) for 0 ) for 0 xx, , y y in Example in Example

3.2.2 3.2.2 (0, 0), a maximum point, failing the SONC(0, 0), a maximum point, failing the SONC

16

SONC for Unconstrained NLPSONC for Unconstrained NLP ff CC22 in S in S xx** an interior local minimum of an interior local minimum of ff, then, then (i).(i). TTff(x(x**) = 0, and ) = 0, and (ii). for (ii). for allall d, d d, dTTH(xH(x**)d )d 0 0 

(ii) H(xH(x**) being positive semi-definite ) being positive semi-definite convex convex f f satisfying (ii) (and actually more)satisfying (ii) (and actually more)

17

Example 3.3.2Example 3.3.2 identity candidates of minimum points for

the f(x) = Tf(x*) = x = (1, -1) or (-1, -1) H(x) = (1, -1) satisfying SONC but not (-1, -1)

3 21 2 1 23 2x x x x

21 1 2(3 3 ,2 2)x x x

16 00 2x

18

SONC Not SufficientSONC Not Sufficient ff((xx, , yy) = -() = -(xx44 + + yy44)) TTff((0, 0))d = 0 for all d((0, 0))d = 0 for all d (0, 0) a maximum(0, 0) a maximum

19

SOSC for Unconstrained NLPSOSC for Unconstrained NLP ff CC22 on S on S nn and and xx** an interior point an interior point if if

(i). (i). TTff(x(x**) = 0, and ) = 0, and (ii). H(x(ii). H(x**) is positive definite) is positive definite

xx** a strict local minimum of a strict local minimum of ff

20

SOSC Not NecessarySOSC Not Necessary Example 3.3.4. x = 0 a minimum of f(x) = x4 SOSC not satisfied

21

Example 3.3.5Example 3.3.5 In Example 3.2.4, is (1, 1, 1) a minimum?

.

6 > 0;

positive definite, i.e., SOSC satisfied

6 2 2( ) 2 2 0

2 0 2

H x

6 212 ( 2)( 2) 8;

2 2

6 2 22 2 0 (6)(2)(2) ( 2)(2)( 2) ( 2)( 2)(2) 82 0 2

2 2 21 2 3 1 2 3 1 2 1 3 1( , , ) 3 2 2 2f x x x x x x x x x x x

22

Effect of ConvexityEffect of Convexity If for all y in the neighborhood of x* S,

Tf(x*)(y-x*) 0 convexity of f implies

f(y) f(x*) + Tf(x*)(y-x*) f(x*) x* a local min of f in the neighborhood of x*

x* a global minimum of f

23

Effect of ConvexityEffect of Convexity f C2 convex H positive semi-definite

everywhere Taylor's Theorem, when Tf(x*)(x-x*) = 0, f(x) = f(x*) + Tf(x*)(x-x*)

+ (x- x*)TH(x* + (1-)x)(x - x*) = f(x*) + (x- x*)TH(x* + (1-)x)(x - x*) f(x*)

x* a local min a global min

24

Effect of ConvexityEffect of Convexity facts of convex functions

 (i). a local min = a global min (ii). H(x) positive semi-definite everywhere (iii). strictly convex function, H(x) positive definite

everywhere implications

for f C2 convex function, the FONC Tf(x*) = 0 is sufficient for x* to be a global minimum

if f strictly convex, x* the unique global min