![Page 1: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/1.jpg)
Introduction to
Convex Constrained Optimization
March 12, 2002
![Page 2: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/2.jpg)
Overview�Nonlinear Optimization
�Portfolio Optimization
�An Inventory Reliability Problem
�Further Concepts for Nonlinear Optimization
�Convex Sets and Convex Functions
�Convex Optimization
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 1
![Page 3: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/3.jpg)
Overview�Pattern Classification
�Some Geometry Problems
�On the Geometry of Nonlinear Optimization
�Classification of Nonlinear Optimization Problems
�Solving Separable Convex Optimization via LinearOptimization
�Optimality Conditions for Nonlinear Optimization
�A Few Concluding Remarks
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 2
![Page 4: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/4.jpg)
Nonlinear versusLinear Optimization
Linear Optimization
�� � ��������� ���
����
���� � ���
�
� �
...
���� � ���
� � ���c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 3
![Page 5: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/5.jpg)
Nonlinear versusLinear Optimization
Nonlinear Optimization
�� � ��������� ��
���� ���� � ��
�
� �
...
���� � ��
� � ���
In this model, we have �� � �� �� �
and ���� � �� �� �� � � � � � � .
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 4
![Page 6: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/6.jpg)
PortfolioOptimization
Investor Preferences
Investors prefer higher annual rates of return oninvesting to lower annual rates of return.
Furthermore, investors prefer lower risk to higher risk.
Portfolio optimization seeks to optimally trade off riskand return in investing.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 5
![Page 7: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/7.jpg)
PortfolioOptimization
Portfolio Fractions
We consider � assets, whose annual rates of return �� arerandom variables, � � �� � � � � �.
The expected annual return of asset � is ��, � � �� � � � � �.
If we invest a fraction �� of our investment dollar in asset �,
the expected return of the portfolio is:
��������� ���
������� ��� �� � � � �
(� � � � � � � �� )
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 6
![Page 8: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/8.jpg)
PortfolioOptimization
Portfolio Covariances & Variance
The covariance of the rates of return of assets � and � is given as
�� � ������ ��
We can think of the �� values as forming a matrix
The variance of portfolio is then:
�����
�����
������ ������ �
�����
����������� � ���
will always by SPD (symmetric positve-definite).
The “risk” in the portfolio is the standard deviation of the portfolio:
��� ��
���c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 7
![Page 9: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/9.jpg)
PortfolioOptimization
Maximal Return Model
We would like to determine the fractional investment values
��� � � � � �� in order to maximize the return of the portfolio, subject
to meeting some pre-specified target risk level, such as ��� �.
MAXIMIZE: RETURN ���
s.t.FSUM: ��� ST. DEV.:
����� � ���
NONNEGATIVITY: � � � .c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 8
![Page 10: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/10.jpg)
PortfolioOptimization
Minimal Risk Model
Alternatively, we might want to determine the fractional
investment values ��� � � � � �� in order to minimize the risk of the
portfolio, subject to meeting a pre-specified target expected return
level, such as ��� �.
MINIMIZE: STDEV �
����
s.t.FSUM: ��� EXP. RETURN: ��� � ���NONNEGATIVITY: � � � �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 9
![Page 11: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/11.jpg)
PortfolioOptimization
Discussion
Variations of this type of model are used pervasivelyby asset management companies worldwide.
This model can be extended to yield the CAPM(Capital Asset Pricing Model).
The Nobel Prize in economics was awarded in 1990 toMerton Miller, William Sharpe, and Harry Markowitz fortheir work on portfolio theory and portfolio models (andthe implications for asset pricing).
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 10
![Page 12: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/12.jpg)
An InventoryReliability Problem
Given positive coefficients ��� ��� ��� � � � � � � , and
Æ � �, solve for ��� � � � � ��:
��� � ���������
���������
����
������������� � �
�� � �� � � � � � �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 11
![Page 13: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/13.jpg)
Further Concepts forNonlinear Optimization
Feasible Region, Ball
Recall the basic nonlinear optimization model:
�� � ��������� ���
���� ����� � �
� �
� �
...
����� � �
� � ���
where ��� � �� �� � and ����� � �� �� �� � � �� � � � ��.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 12
![Page 14: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/14.jpg)
Further Concepts forNonlinear Optimization
Feasible Region, Ball
The feasible region of �� is the set:
������ � �� � � � � ���� � ��
The ball centered at �� with radius � is the set:
���� �� � �� �� �� � ��
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 13
![Page 15: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/15.jpg)
Further Concepts forNonlinear Optimization
Local & Global Minima
Definition 1 � � is a local minimum of �� ifthere exists � � � such that �� � �� for all
� � ��� �� � .
Definition 2 � � is a global minimum of �� if
�� � �� for all � � .
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 14
![Page 16: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/16.jpg)
Further Concepts forNonlinear Optimization
Strict Local & Global Minima
Definition 3 � � is a strict local minimum of ��
if there exists � � � such that �� � �� for all
� � ��� �� � , � � �.
Definition 4 � � is a strict global minimum of
�� if �� � �� for all � � , � � �.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 15
![Page 17: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/17.jpg)
Further Concepts forNonlinear Optimization
Local & Global Maxima
Definition 5 � � is a local maximum of �� ifthere exists � � � such that �� � �� for all
� � ��� �� � .
Definition 6 � � is a global maximum of �� if
�� � �� for all � � .
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 16
![Page 18: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/18.jpg)
Further Concepts forNonlinear Optimization
Strict Local & Globla Maxima
Definition 7 � � is a strict local maximum of
�� if there exists � � � such that �� � �� forall � � ��� �� � , � � �.
Definition 8 � � is a strict global maximum of
�� if �� � �� for all � � , � � �.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 17
![Page 19: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/19.jpg)
Further Concepts forNonlinear Optimization
Local versus Global Minima
F
Illustration of local versus global optima.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 18
![Page 20: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/20.jpg)
Further Concepts forNonlinear Optimization
Convex Sets and Functions
Definition 9 A subset � � �� is a convex set if
�� � � � � ��� � ��� � �
for any � � ��� �.
Illustration of convex and non-convex sets.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 19
![Page 21: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/21.jpg)
Further Concepts forNonlinear Optimization
Intersections of Convex Sets
Proposition 1 If ��� are convex sets, then � � � isa convex set.
Proposition 2 The intersection of any collection ofconvex sets is a convex set.
Illustration of the intersection of convex sets.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 20
![Page 22: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/22.jpg)
Further Concepts forNonlinear Optimization
Convex Functions
Definition 10 A function �� is a convex function if��� � ���� � ��� � � ����
for all � and � and for all � � ��� �.
Definition 11 A function �� is a strictly convexfunction if
��� � ���� � ��� � � ����
for all � and � and for all � � �� �, � � �.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 21
![Page 23: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/23.jpg)
Further Concepts forNonlinear Optimization
Convex Functions
Illustration of convex and strictly convex functions.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 22
![Page 24: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/24.jpg)
Further Concepts forNonlinear Optimization
Convex Optimization
� � ��������� ��
����
� �
Proposition 3 Suppose that is a convex set,
� � � is a convex function, and �� is a localminimum of � . Then �� is a global minimum of over
.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 23
![Page 25: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/25.jpg)
Further Concepts forNonlinear Optimization
Proof of Proposition
Proof of the Proposition: Suppose �� is not a global minimum,i.e., there exists � � � for which ��� � ����. Let
���� � ���������, which is a convex combination of �� and �
for � � � ��� (and therefore, ���� � � for � � � ���). Note that
����� �� as �� �.
From the convexity of ���,
������ � ����� �� ���� � � ���� � �� �� ��� � � ���� � �� �� ���� � ����
for all � � � ���. Therefore, ������ � ���� for all � � � ���,
and so �� is not a local minimum, resulting in a contradiction.
q.e.d.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 24
![Page 26: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/26.jpg)
Further Concepts forNonlinear Optimization
Examples of Convex Functions
Functions of One Variable
Examples of convex functions of one variable:
��� ��� �
��� ��� ��� �
��� ���
��� � ���� for � � �
��� ��
for � � �
��� ��
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 25
![Page 27: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/27.jpg)
Further Concepts forNonlinear Optimization
Concave Functions & Maximization
The “opposite” of a convex function is a concave function:
Definition 12 A function �� is a concave function if
��� � ���� � ��� � � ����
for all � and � and for all � � ��� �.
Definition 13 A function �� is a strictly concavefunction if
��� � ���� � ��� � � ����
for all � and � and for all � � �� �, � � �.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 26
![Page 28: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/28.jpg)
Further Concepts forNonlinear Optimization
Concave Functions & Maximization
Illustration of concave and strictly concave functions.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 27
![Page 29: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/29.jpg)
Further Concepts forNonlinear Optimization
Concave Maximization
Now consider the maximization problem � :
� � ��������� ��
����
� �
Proposition 4 Suppose that is a convex set,
� � � is a concave function, and �� is a localmaximum of � . Then �� is a global maximum of over
.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 28
![Page 30: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/30.jpg)
Further Concepts forNonlinear Optimization
Linear Functions
Proposition 5 A linear function �� ���� � isboth convex and concave.
Proposition 6 If �� is both convex and concave,then �� is a linear function.
A linear function is convex and concave.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 29
![Page 31: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/31.jpg)
Further Concepts forNonlinear Optimization
Convex Optimization
Level Sets of Convex Functions...
Suppose that �� is a convex function. The set
�� � ���� � ��
is the level set of �� at level �.
Proposition 7 If �� is a convex function, then ��
is a convex set.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 30
![Page 32: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/32.jpg)
Further Concepts forNonlinear Optimization
Convex Optimization
...Level Sets of Convex Functions
The level sets of a convex function are convex sets.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 31
![Page 33: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/33.jpg)
Further Concepts forNonlinear Optimization
Convex Optimization
Convex Optimization Model...
� � ��������� ��
���� ���� � ��
...
���� � ��
!� ��
� � ���
�� is called a convex optimization problem
if ���� ������ � � � � ����� are convex functions.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 32
![Page 34: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/34.jpg)
Further Concepts forNonlinear Optimization
Convex Optimization
...Convex Optimization Model...
Proposition 8 The feasible region of � is aconvex set.
Corollary 1 Any local minimum of � will be aglobal minimum of � .
This is a most important aspect of a convexoptimization problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 33
![Page 35: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/35.jpg)
Further Concepts forNonlinear Optimization
Convex Optimization
...Convex Optimization Model
Remark 1 If we replace “min” by “max” in � and if
�� is a concave function while ����� � � � � ���� areconvex functions, then any local maximum of � willbe a global maximum of � .
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 34
![Page 36: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/36.jpg)
Further Concepts forNonlinear Optimization
Further Properties of Convex Functions
Additions & Affine Transformations
Proposition 9 If ��� and ��� are convexfunctions, and �� � � �, then
�� � ���� � ����
is a convex function.
Proposition 10 If �� is a convex function and
� !�� �, then
��� � !�� ��
is a convex function.c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 35
![Page 37: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/37.jpg)
Further Concepts forNonlinear Optimization
Further Properties of Convex Functions
Gradient and Hessian
A function �� is twice differentiable at � �� if thereexists a vector ���� (called the gradient of ��) anda symmetric matrix"��� (called the Hessian of ��)for which:
�� ��� �������� ��� � ���� ���
�"����� ��� �#���� �� �� �
where #����� � as �� ��.
���� �$���
$��� � � � �$���
$���
� "���� $����
$��$��
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 36
![Page 38: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/38.jpg)
Further Concepts forNonlinear Optimization
Further Properties of Convex Functions
Gradient & Hessian of Convex Functions
Theorem 1 Suppose that �� is twice differentiableon the open convex set �. Then �� is a convexfunction on the domain � if and only if "�� is SPSDfor all � � �.
(Recall that SPSD means symmetric and positivesemi-definite.)
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 37
![Page 39: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/39.jpg)
Further Concepts forNonlinear Optimization
Examples of Convex Functions
Examples in � Dimensions��� ���� �
��� ������� ��� where� is SPSD
��� � for any norm �
���
������ ���� � ��
� �� for � satisfying !� � �.
Corollary 2 If �� ������� ��� where � is a
symmetric matrix, then �� is a convex function if andonly if � is SPSD. Furthermore, �� is a strictlyconvex function if and only if � is SPD.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 38
![Page 40: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/40.jpg)
Further Concepts forNonlinear Optimization
Examples of Convex Functions
Norm Function
Illustration of a norm function.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 39
![Page 41: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/41.jpg)
Further Concepts forNonlinear Optimization
Examples of Convex Functions
The Gradient Inequality
The gradient inequality for convex functions:
Proposition 11 If �� is a differentiable convexfunction, then for any �� �,
�� � �� ������ � �� �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 40
![Page 42: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/42.jpg)
Further Concepts forNonlinear Optimization
Examples of Convex Functions
The Subgradient Inequality
Even when a convex function is not differentiable, itmust satisfy the following “subgradient inequality.”
Proposition 12 If �� is a convex function, then forevery �, there must exist some vector � for which
�� � �� � ��� � �� for any � �
The vector � in this proposition is called a subgradientof �� at the point �.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 41
![Page 43: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/43.jpg)
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem
We are given:
� points ��� � � � � �� � �� that have property “P”
� points ��� � � � � �� � �� that do not have property “P”
Illustration of the pattern classification problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 42
![Page 44: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/44.jpg)
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem
We seek a vector % and a scalar � for which:
�%��� � � for all � � � � � � &
�%��� � � for all � � � � � �
We will then use %� � to predict whether or not anyother point � has property P or not.
� If %�� � �, then we declare that � has property P.
� If %�� � �, then we declare that � does not haveproperty P.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 43
![Page 45: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/45.jpg)
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem%� � define the hyperplane
"� � � ��%�� ��
for which:
�%��� � � for all � � � � � � &
�%��� � � for all � � � � � � We would like the hyperplane"� � to be as far awayfrom the points ��� � � � � ��� ��� � � � � �� as possible.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 44
![Page 46: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/46.jpg)
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem
The distance of the hyperplane ��� to any point �� is
���� ��
The distance of the hyperplane ��� to any point �� is
� ����
�If we normalize the vector � so that � � �, then the minimimdistance of the hyperplane ��� to the points ��� � � � � ��
��� � � � � � is:
�������� �� � � � � ��� ��� ����� � � � � � �����
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 45
![Page 47: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/47.jpg)
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem
We would also like � and � to satisfy:
� � �, and
�������� �� � � � � ��� ��� ����� � � � � � ����� ismaximized.
��� � �����������Æ Æ
����
���� � � � � � �� � � � � �
� ���� � � � � �� � � � ��
� � ��
� � ��� � � �c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 46
![Page 48: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/48.jpg)
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem� � � ��������� � Æ Æ
����
%��� � � � � � � � � � � &
� � %��� � � � � � � � �
% �
% � ��� � � �
� � is not a convex optimization problem,due to the presence of the constraint “ % .”
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 47
![Page 49: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/49.jpg)
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem
��� � ���������Æ Æ
�� � ���� � � � Æ � � � � � � �
� � ���� � Æ � � � � � � �
��� � �
� � �� � � �
Transformation of variables:� %
Æ
� � �
Æ
��� Æ ���� %
� ����
� ���� �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 48
![Page 50: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/50.jpg)
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem
��� � ���������Æ Æ
s.t. ���� � � � Æ � � � � � � �
� � ���� � Æ � � � � � � �
��� � �
� � �� � � �
� ��
Æ
� ��
Æ
���� � ����������� �
����
���� � � �� � � �� � � � � �
� ���� � �� � � �� � � � ��
� � ��� � � �c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 49
![Page 51: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/51.jpg)
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem
���� � ������� ���
�� �
���� �� � � � � � � � � �
�� ���� � � � � � � � � �
� � �� � � �
� � is a convex problem.
Solve � � for ���, and substitute
% �
� �
� �
� �
Æ
�
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 50
![Page 52: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/52.jpg)
More Examples of ConvexOptimization Problems
The Minimum Norm Problem
Given a vector �, we would like to find the closest pointto � that also satisfies the linear inequalities !� � �.
�� � ��������� �� �
����
!� � �
� � ��
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 51
![Page 53: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/53.jpg)
More Examples of ConvexOptimization Problems
The Minimum Norm Problem
Illustration of the minimum norm problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 52
![Page 54: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/54.jpg)
More Examples of ConvexOptimization Problems
The Fermat-Weber Problem
Given points ��� � � � � �� � ��, determine thelocation of a distribution center at the point � � �� thatminimizes the sum of the distances from � to each ofthe points ��� � � � � �� � ��.
'(� � �����������
��� �� ��
����
� � ��
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 53
![Page 55: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/55.jpg)
More Examples of ConvexOptimization Problems
The Fermat-Weber Problem
Illustration of the Fermat-Weber problem.
'(� is a convex unconstrained problem.c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 54
![Page 56: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/56.jpg)
More Examples of ConvexOptimization Problems
The Ball Circumscription Problem
Given points ��� � � � � �� � ��, determine thelocation of a distribution center at the point � � �� thatminimizes the maximum distance from � to any of thepoints ��� � � � � �� � ��.
� � � ��������� Æ Æ
����
�� �� � � � � � � � � �
� � ��
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 55
![Page 57: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/57.jpg)
More Examples of ConvexOptimization Problems
The Ball Circumscription Problem
Illustration of the ball circumscription problem.
This is a convex (constrained) optimization problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 56
![Page 58: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/58.jpg)
More Examples of ConvexOptimization Problems
The Analytic Center Problem
Given a system of linear inequalities !� � �, wewould like to determine a “nicely” interior point �� thatsatisfies !�� � �.
! � � �����������
�����!���
����
!� � ��
� � ��c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 57
![Page 59: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/59.jpg)
More Examples of ConvexOptimization Problems
The Analytic Center Problem
Illustration of the analytic center problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 58
![Page 60: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/60.jpg)
More Examples of ConvexOptimization Problems
The Analytic Center Problem
��� � ������
�����������
�� �
�� � �
� � ��
Proposition 13 Suppose that �� solves the analyticcenter problem ! � . Then for each �, we have:
��!���� ����
where
��� � ���
������!��� �c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 59
![Page 61: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/61.jpg)
More Examples of ConvexOptimization Problems
The Analytic Center Problem
��� � ������
�����������
�� �
�� � �
� � ��
��� is not a convex problem, but it is equivalent to:
���� � �����������
��� ����������
����
�� � ��
� � ��
Now notice that ���� is a convex problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 60
![Page 62: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/62.jpg)
More Examples of ConvexOptimization Problems
The Circumscribed Ellipsoid Problem
Given � points ��� � � � � �� � ��, determine an ellipsoid of
minimum volume that contains each of the points
��� � � � � �� � ��.
Illustration of the circumscribed ellipsoid problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 61
![Page 63: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/63.jpg)
More Examples of ConvexOptimization Problems
The Circumscribed Ellipsoid Problem
Recall that an SPD matrix � and a given point � can be used todefine an ellipsoid in ��:
��� �� �� � �� ������ �� � ���
The volume of ��� is proportional to ���������
�.
Illustration of an ellipsoid.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 62
![Page 64: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/64.jpg)
More Examples of ConvexOptimization Problems
The Circumscribed Ellipsoid Problem
Our problem is:� �� � �������� ���#����
�
#�)���� �� � *� �� � � � � � � &�
# is SPD.
� �� � �������� � �����#��
#�)���� �� � )��#�� � )� � � � � � � � � &
# is SPD.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 63
![Page 65: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/65.jpg)
More Examples of ConvexOptimization Problems
The Circumscribed Ellipsoid Problem
���� � ���� � ����� ����
� �
�� � ��� � ������� � �� � � � � � � � � �
� is SPD.
Factor # �� where � is SPD
� �� � �������� � ���������
��)���� �� � )������� � )� � � � � � � � � &�
� is SPD.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 64
![Page 66: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/66.jpg)
More Examples of ConvexOptimization Problems
The Circumscribed Ellipsoid Problem
���� � ���� � ����� �����
� �
�� � ��� � ��������� � �� � � � � � � � � �
� is SPD.
� �� � �������� �� ��������
��)
���� ��� � )� � � � � � � � � &�
� is SPD.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 65
![Page 67: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/67.jpg)
More Examples of ConvexOptimization Problems
The Circumscribed Ellipsoid Problem
���� � ���� �� ����� ����
� �
�� � ����� � ��� � � � � � � � � �
� is SPD.
Next substitute � ��� to obtain:
���� � �������� ����������
���
���� ��� � � �� � � �� � � � � ��
� is SPD.– This is convex problem in the variables ���.
– We can recover � and � after solving ���� by substituting
� ��� and � �����.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 66
![Page 68: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/68.jpg)
Classification of Nonlinear Optimization Problems
General Nonlinear Problem��� or ��� ��
���� ���� � �� � � � � � � �
Unconstrained Optimization
��� or ��� ��
���� � � ��c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 67
![Page 69: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/69.jpg)
Classification of Nonlinear Optimization Problems
Linearly Constrained Problems
��� or ��� ��
���� !� � �
Linear Optimization��� ���
���� !� � �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 68
![Page 70: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/70.jpg)
Classification of Nonlinear Optimization Problems
Quadratic Problem
��� ���� ������
���� !� � �
The objective function here is
�����
���� �
����
����������.
Quadratically Constrained Quadratic Problem
��� ���� ������
���� ��� ���
������ � �� � � � � � � �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 69
![Page 71: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/71.jpg)
Classification of Nonlinear Optimization Problems
Separable Problem
���
����� ����
����
������������ � �� � � � �� � � � ��
Example of a separable problem
��� ��� � ��
� � ��
���� ��� ��� !"�� � #
������
� �� ������ � � �$
� � � � � � � � c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 70
![Page 72: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/72.jpg)
Classification of Nonlinear Optimization Problems
Convex Problem
��� ���
���� ����� � �� � � � �� � � � ��
�� � �
where ��� is a convex function and ������ � � � � ����� areconvex functions, or:
��� ���
���� ����� � �� � � � �� � � � ��
�� � �
where ��� is a concave function and ������ � � � � ����� are
convex functions.c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 71
![Page 73: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/73.jpg)
Solving Separable Convex Optimization via Linear Optimization
Consider the problem
��� ��� ��� �� � #��
���� ���� ���%�� � &
&���%��� �� � $
��� ��� �� �
This problem has:
1. linear constraints
2. a separable objective function � � ����� � ����� � #��
3. a convex objective function (since ���� and ���� are convexfunctions)
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 72
![Page 74: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/74.jpg)
Solving Separable Convex Optimization via Linear Optimization
��� �� � ���� �������
���� ���� ���� ��� � �
���� ���� �� � �
��� ��� �� � �
We can approximate ��� and ��� bypiecewise-linear (PL) functions.
Suppose we know that � � �� � � and � � �� � �.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 73
![Page 75: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/75.jpg)
Solving Separable Convex Optimization via Linear Optimization
PL approximation of ����.
����� � ��� �� � ' ����%���� �����
�� � ���� ���� ���
� ��� � �� � ��� � �� � ��� � � �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 74
![Page 76: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/76.jpg)
Solving Separable Convex Optimization via Linear Optimization
PL approximation of ����.
����� � �� � � � ������ �$������ �'#�����
�� � ���� ���� ���
� ��� � �� � ��� � �� � ��� � �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 75
![Page 77: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/77.jpg)
Solving Separable Convex Optimization via Linear Optimization
The linear optimization approximation then is:
��� �� ���� ����� ���� ��
����� ��
����� ���
����� ���
���� ���� ���� ��� � �
���� ���� �� � �
��� ��� �� � �
���� ���� ��� ��� ���� ���� ��� ��
� � ��� � �� � � ��� � �� � � ��� � �
� � ��� � �� � � ��� � �� � � ��� � �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 76
![Page 78: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/78.jpg)
On the Geometry ofNonlinear Optimization
Extreme Point Solution
First example of the geometry of the solutionof a nonlinear optimization problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 77
![Page 79: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/79.jpg)
On the Geometry ofNonlinear Optimization
Non-Extreme Point Boundary Solution
Second example of the geometry of the solutionof a nonlinear optimization problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 78
![Page 80: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/80.jpg)
On the Geometry ofNonlinear Optimization
Interior Solution
Third example of the geometry of the solutionof a nonlinear optimization problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 79
![Page 81: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/81.jpg)
On the Geometry ofNonlinear Optimization
Discussion
Unlike linear optimization, the optimal solution of anonlinear optimization problem need not be a “cornerpoint” of .
The optimal solution may be on the boundary or evenin the interior of .
The optimal solution will be characterized by how thecontours of �� are “aligned” with the feasible region
.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 80
![Page 82: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/82.jpg)
Optimality Conditions forNonlinear Optimization
Convex Problem
Consider the convex problem:
� � � ��� ��
���� ���� � �� � � � � � � � �
��� ���� are convex functions.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 81
![Page 83: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/83.jpg)
Optimality Conditions forNonlinear Optimization
The KKT Theorem
Convex Optimization
Theorem 2 (Karush-Kuhn-Tucker Theorem)Suppose that ��� ����� � � � � ���� are all convexfunctions. Then under very mild conditions, �� solves(CP) if and only if there exists ��� � �� � � � � � � ,such that
(�) ���� ���
������������ � (gradients line up)
(��) ������ �� � � (feasibility)
(���) ��� � � (multipliers must be nonnegative)
(��) ����� � ������ �� (multipliers for nonbinding constraints are zero)
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 82
![Page 84: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/84.jpg)
Optimality Conditions forNonlinear Optimization
The KKT Theorem
Linear Optimization & Strong Duality...
Consider the following linear optimization problem andits dual:
! � � ��� ��� !"� � �����
��������
��� � � �� � � � � � � � ���
������� �
� � �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 83
![Page 85: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/85.jpg)
Optimality Conditions forNonlinear Optimization
The KKT Theorem
...Linear Optimization & Strong Duality...
Let us look at the KKT Theorem applied here. If ��
solves LP, then there exists ���� � � � � � � for which:
(�) ���� ���
������������ � ��
���������� � (gradients line up)
(��) ������ �� � ��� �� �� � � ( primal feasibility)
(���) ��� � �
(��) ����� � ������ � ����� � ��
� �� �� (complementary slackness)
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 84
![Page 86: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/86.jpg)
Optimality Conditions forNonlinear Optimization
The KKT Theorem
...Linear Optimization & Strong Duality
(�) ������ ��
������������� �� ��
��������� � � (gradients line up)
(��) ������� �� �� ��� �� �� � � ( primal feasibility)
(���) ��� � �
(��) ������ � ������� �� ������ � ��� �� � �� (complementary slackness)
– (��) is primal feasbility of ��– (�) and (���) together are dual feasibility of ��
– (��) is complementary slackness
The KKT Theorem states that �� and �� together must be
primal feasible and dual feasible, and must satisfy complementary
slackness.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 85
![Page 87: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/87.jpg)
Optimality Conditions forNonlinear Optimization
The KKT Theorem
Geometry
g1(x) = 0
g2(x) = 0
g3(x) = 0
∇g1(x)
∇g2(x)
−∇f(x)
∇f(x) x–
Illustration of the KKT Theorem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 86
![Page 88: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/88.jpg)
Optimality Conditions forNonlinear Optimization
The KKT Theorem
An example...
Consider the problem:
��� ���� � �� �'��� � �%��
���� ���
���� %�� � %
���
�����
�
��� ��� ����
� �&
��� � ���� � ���'��� � �%��
����� � ���
� ��� %�� %
����� � ���
�����
����� � ��� ���� ���
�&
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 87
![Page 89: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/89.jpg)
Optimality Conditions forNonlinear Optimization
The KKT Theorem
...An example...
�� ���� � ���� ����� � ������
�� � ���
���� � ��� � ��
���
�����
� ���
��� � ��� ����
� ��
� ��� ��
�� ��� � �
$��� � �%��
�
������ ��
� ��
��� %��
� ������ ��
� ��
����
� ������ ��
� ��� ��
��
��
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 88
![Page 90: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/90.jpg)
Optimality Conditions forNonlinear Optimization
The KKT Theorem
...An example...
��� ��� � ��� ���� � �����
���� ��� ��� � ��� � ��
��� ����� � ���
�� � ��� ���� � ��
Is �� ���� ���� ���� an optimal solution?
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 89
![Page 91: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/91.jpg)
Optimality Conditions forNonlinear Optimization
The KKT Theorem
...An example...
�� ���� � ���� ����� � ������
�� � ���
���� � ��� � ��
���
�����
� ���
��� � ��� ����
� ��
We first check for feasibility of �� ����:����� �
����� ��� � �
����� �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 90
![Page 92: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/92.jpg)
Optimality Conditions forNonlinear Optimization
The KKT Theorem
...An example...
�� ���� � ���� ����� � ������
�� � ���
���� � ��� � ��
���
�����
� ���
��� � ��� ����
� ��
To check for optimality, we compute all gradients at ��:
��� �
�������
�����
�� �
�
����� �
� ���
�����
�� �
�
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 91
![Page 93: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/93.jpg)
Optimality Conditions forNonlinear Optimization
The KKT Theorem
...An example
�� ���� � ���� ����� � ������
�� � ���
���� � ��� � ��
���
�����
� ���
��� � ��� ����
� ��
We next try to solve for �� � � �� � � �� � in the system:
����
% �
���
��'
�
�����
��'
���
�����
�
� �
��� ��
�
�
�
�� � ����� ���� ���� � � � �'� solves this system in nonnegativevalues, and �� � .
Therefore �� is an optimal solution to this problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 92
![Page 94: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/94.jpg)
Optimality Conditions forNonlinear Optimization
KKT Theorem with Different Formats of Constraints
Convex Optimization with Linear Equations
����� � �� ����
�� � ����� � �� � � �
��� � � �� � � �
Theorem 3 (Karush-Kuhn-Tucker Theorem) Suppose that ���� ����� are allconvex functions for � � �. Then under very mild conditions, �� solves (CPE) ifand only if there exists ��� � � � � � and ��� � � � such that
(�) ������ ��
������������ �
�����
����� � � (gradients line up)
(����) ������� �� � � � � � (feasibility)
(����) ��� �� �� � � � � � (feasibility)
(���) ��� � � � � � (nonnegative multipliers on inequalities)
(��) ������ � ������� � � � � �� (complementary slackness)
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 93
![Page 95: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/95.jpg)
Optimality Conditions forNonlinear Optimization
KKT Theorem with Different Formats of Constraints
Non-convex Optimization
����� � �� ����
�� � ����� � �� � � �
����� � �� � � �
Theorem 4 (Karush-Kuhn-Tucker Theorem) Under some very mild conditions,if �� solves (NCE), then there exists ��� � � � � � and ��� � � � such that
(�) ������ ��
������������ �
����
��������� � � (gradients line up)
(����) ������� �� � � � � � (feasibility)
(����) ������ �� � � � � � (feasibility)
(��� ) ��� � � (nonnegative multipliers on inequalities)
(��) ������ � ������� � �� (complementary slackness)
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 94
![Page 96: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/96.jpg)
A Few Concluding Remarks
Nonconvex optimization problems can be difficult tosolve.
This is because local optima may not be global optima.
Most algorithms are based on calculus, and so canonly find a local optimum.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 95
![Page 97: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/97.jpg)
A Few Concluding Remarks
There are a large number of solution methods forsolving nonlinear constrained problems.
Today, the most powerful methods fall into two classes:
�methods that try to generalize the simplex algorithmto the nonlinear case.
�methods that generalize the barrier method to thenonlinear case.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 96
![Page 98: Introduction to Convex Constrained Optimizationdspace.mit.edu › ... › 0 › lecture10.pdf · 2019-09-12 · Nonlinear Optimization Proof of Proposition Proof of the Proposition:](https://reader034.vdocument.in/reader034/viewer/2022042311/5ed932b76714ca7f47695379/html5/thumbnails/98.jpg)
A Few Concluding Remarks
Quadratic Problems. A quadratic problem is “almostlinear”, and can be solved by a special implementationof the simplex algorithm, called the complementarypivoting algorithm.
Quadratic problems are roughly eight times harder tosolve than linear optimization problems of comparablesize.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 97