Introduction to
Convex Constrained Optimization
March 12, 2002
Overview�Nonlinear Optimization
�Portfolio Optimization
�An Inventory Reliability Problem
�Further Concepts for Nonlinear Optimization
�Convex Sets and Convex Functions
�Convex Optimization
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 1
Overview�Pattern Classification
�Some Geometry Problems
�On the Geometry of Nonlinear Optimization
�Classification of Nonlinear Optimization Problems
�Solving Separable Convex Optimization via LinearOptimization
�Optimality Conditions for Nonlinear Optimization
�A Few Concluding Remarks
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 2
Nonlinear versusLinear Optimization
Linear Optimization
�� � ��������� ���
����
���� � ���
�
� �
...
���� � ���
� � ���c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 3
Nonlinear versusLinear Optimization
Nonlinear Optimization
�� � ��������� ��
���� ���� � ��
�
� �
...
���� � ��
� � ���
In this model, we have �� � �� �� �
and ���� � �� �� �� � � � � � � .
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 4
PortfolioOptimization
Investor Preferences
Investors prefer higher annual rates of return oninvesting to lower annual rates of return.
Furthermore, investors prefer lower risk to higher risk.
Portfolio optimization seeks to optimally trade off riskand return in investing.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 5
PortfolioOptimization
Portfolio Fractions
We consider � assets, whose annual rates of return �� arerandom variables, � � �� � � � � �.
The expected annual return of asset � is ��, � � �� � � � � �.
If we invest a fraction �� of our investment dollar in asset �,
the expected return of the portfolio is:
��������� ���
������� ��� �� � � � �
(� � � � � � � �� )
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 6
PortfolioOptimization
Portfolio Covariances & Variance
The covariance of the rates of return of assets � and � is given as
�� � ������ ��
We can think of the �� values as forming a matrix
The variance of portfolio is then:
�����
�����
������ ������ �
�����
����������� � ���
will always by SPD (symmetric positve-definite).
The “risk” in the portfolio is the standard deviation of the portfolio:
��� ��
���c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 7
PortfolioOptimization
Maximal Return Model
We would like to determine the fractional investment values
��� � � � � �� in order to maximize the return of the portfolio, subject
to meeting some pre-specified target risk level, such as ��� �.
MAXIMIZE: RETURN ���
s.t.FSUM: ��� ST. DEV.:
����� � ���
NONNEGATIVITY: � � � .c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 8
PortfolioOptimization
Minimal Risk Model
Alternatively, we might want to determine the fractional
investment values ��� � � � � �� in order to minimize the risk of the
portfolio, subject to meeting a pre-specified target expected return
level, such as ��� �.
MINIMIZE: STDEV �
����
s.t.FSUM: ��� EXP. RETURN: ��� � ���NONNEGATIVITY: � � � �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 9
PortfolioOptimization
Discussion
Variations of this type of model are used pervasivelyby asset management companies worldwide.
This model can be extended to yield the CAPM(Capital Asset Pricing Model).
The Nobel Prize in economics was awarded in 1990 toMerton Miller, William Sharpe, and Harry Markowitz fortheir work on portfolio theory and portfolio models (andthe implications for asset pricing).
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 10
An InventoryReliability Problem
Given positive coefficients ��� ��� ��� � � � � � � , and
Æ � �, solve for ��� � � � � ��:
��� � ���������
���������
����
������������� � �
�� � �� � � � � � �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 11
Further Concepts forNonlinear Optimization
Feasible Region, Ball
Recall the basic nonlinear optimization model:
�� � ��������� ���
���� ����� � �
� �
� �
...
����� � �
� � ���
where ��� � �� �� � and ����� � �� �� �� � � �� � � � ��.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 12
Further Concepts forNonlinear Optimization
Feasible Region, Ball
The feasible region of �� is the set:
������ � �� � � � � ���� � ��
The ball centered at �� with radius � is the set:
���� �� � �� �� �� � ��
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 13
Further Concepts forNonlinear Optimization
Local & Global Minima
Definition 1 � � is a local minimum of �� ifthere exists � � � such that �� � �� for all
� � ��� �� � .
Definition 2 � � is a global minimum of �� if
�� � �� for all � � .
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 14
Further Concepts forNonlinear Optimization
Strict Local & Global Minima
Definition 3 � � is a strict local minimum of ��
if there exists � � � such that �� � �� for all
� � ��� �� � , � � �.
Definition 4 � � is a strict global minimum of
�� if �� � �� for all � � , � � �.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 15
Further Concepts forNonlinear Optimization
Local & Global Maxima
Definition 5 � � is a local maximum of �� ifthere exists � � � such that �� � �� for all
� � ��� �� � .
Definition 6 � � is a global maximum of �� if
�� � �� for all � � .
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 16
Further Concepts forNonlinear Optimization
Strict Local & Globla Maxima
Definition 7 � � is a strict local maximum of
�� if there exists � � � such that �� � �� forall � � ��� �� � , � � �.
Definition 8 � � is a strict global maximum of
�� if �� � �� for all � � , � � �.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 17
Further Concepts forNonlinear Optimization
Local versus Global Minima
F
Illustration of local versus global optima.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 18
Further Concepts forNonlinear Optimization
Convex Sets and Functions
Definition 9 A subset � � �� is a convex set if
�� � � � � ��� � ��� � �
for any � � ��� �.
Illustration of convex and non-convex sets.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 19
Further Concepts forNonlinear Optimization
Intersections of Convex Sets
Proposition 1 If ��� are convex sets, then � � � isa convex set.
Proposition 2 The intersection of any collection ofconvex sets is a convex set.
Illustration of the intersection of convex sets.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 20
Further Concepts forNonlinear Optimization
Convex Functions
Definition 10 A function �� is a convex function if��� � ���� � ��� � � ����
for all � and � and for all � � ��� �.
Definition 11 A function �� is a strictly convexfunction if
��� � ���� � ��� � � ����
for all � and � and for all � � �� �, � � �.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 21
Further Concepts forNonlinear Optimization
Convex Functions
Illustration of convex and strictly convex functions.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 22
Further Concepts forNonlinear Optimization
Convex Optimization
� � ��������� ��
����
� �
Proposition 3 Suppose that is a convex set,
� � � is a convex function, and �� is a localminimum of � . Then �� is a global minimum of over
.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 23
Further Concepts forNonlinear Optimization
Proof of Proposition
Proof of the Proposition: Suppose �� is not a global minimum,i.e., there exists � � � for which ��� � ����. Let
���� � ���������, which is a convex combination of �� and �
for � � � ��� (and therefore, ���� � � for � � � ���). Note that
����� �� as �� �.
From the convexity of ���,
������ � ����� �� ���� � � ���� � �� �� ��� � � ���� � �� �� ���� � ����
for all � � � ���. Therefore, ������ � ���� for all � � � ���,
and so �� is not a local minimum, resulting in a contradiction.
q.e.d.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 24
Further Concepts forNonlinear Optimization
Examples of Convex Functions
Functions of One Variable
Examples of convex functions of one variable:
��� ��� �
��� ��� ��� �
��� ���
��� � ���� for � � �
��� ��
for � � �
��� ��
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 25
Further Concepts forNonlinear Optimization
Concave Functions & Maximization
The “opposite” of a convex function is a concave function:
Definition 12 A function �� is a concave function if
��� � ���� � ��� � � ����
for all � and � and for all � � ��� �.
Definition 13 A function �� is a strictly concavefunction if
��� � ���� � ��� � � ����
for all � and � and for all � � �� �, � � �.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 26
Further Concepts forNonlinear Optimization
Concave Functions & Maximization
Illustration of concave and strictly concave functions.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 27
Further Concepts forNonlinear Optimization
Concave Maximization
Now consider the maximization problem � :
� � ��������� ��
����
� �
Proposition 4 Suppose that is a convex set,
� � � is a concave function, and �� is a localmaximum of � . Then �� is a global maximum of over
.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 28
Further Concepts forNonlinear Optimization
Linear Functions
Proposition 5 A linear function �� ���� � isboth convex and concave.
Proposition 6 If �� is both convex and concave,then �� is a linear function.
A linear function is convex and concave.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 29
Further Concepts forNonlinear Optimization
Convex Optimization
Level Sets of Convex Functions...
Suppose that �� is a convex function. The set
�� � ���� � ��
is the level set of �� at level �.
Proposition 7 If �� is a convex function, then ��
is a convex set.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 30
Further Concepts forNonlinear Optimization
Convex Optimization
...Level Sets of Convex Functions
The level sets of a convex function are convex sets.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 31
Further Concepts forNonlinear Optimization
Convex Optimization
Convex Optimization Model...
� � ��������� ��
���� ���� � ��
...
���� � ��
!� ��
� � ���
�� is called a convex optimization problem
if ���� ������ � � � � ����� are convex functions.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 32
Further Concepts forNonlinear Optimization
Convex Optimization
...Convex Optimization Model...
Proposition 8 The feasible region of � is aconvex set.
Corollary 1 Any local minimum of � will be aglobal minimum of � .
This is a most important aspect of a convexoptimization problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 33
Further Concepts forNonlinear Optimization
Convex Optimization
...Convex Optimization Model
Remark 1 If we replace “min” by “max” in � and if
�� is a concave function while ����� � � � � ���� areconvex functions, then any local maximum of � willbe a global maximum of � .
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 34
Further Concepts forNonlinear Optimization
Further Properties of Convex Functions
Additions & Affine Transformations
Proposition 9 If ��� and ��� are convexfunctions, and �� � � �, then
�� � ���� � ����
is a convex function.
Proposition 10 If �� is a convex function and
� !�� �, then
��� � !�� ��
is a convex function.c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 35
Further Concepts forNonlinear Optimization
Further Properties of Convex Functions
Gradient and Hessian
A function �� is twice differentiable at � �� if thereexists a vector ���� (called the gradient of ��) anda symmetric matrix"��� (called the Hessian of ��)for which:
�� ��� �������� ��� � ���� ���
�"����� ��� �#���� �� �� �
where #����� � as �� ��.
���� �$���
$��� � � � �$���
$���
� "���� $����
$��$��
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 36
Further Concepts forNonlinear Optimization
Further Properties of Convex Functions
Gradient & Hessian of Convex Functions
Theorem 1 Suppose that �� is twice differentiableon the open convex set �. Then �� is a convexfunction on the domain � if and only if "�� is SPSDfor all � � �.
(Recall that SPSD means symmetric and positivesemi-definite.)
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 37
Further Concepts forNonlinear Optimization
Examples of Convex Functions
Examples in � Dimensions��� ���� �
��� ������� ��� where� is SPSD
��� � for any norm �
���
������ ���� � ��
� �� for � satisfying !� � �.
Corollary 2 If �� ������� ��� where � is a
symmetric matrix, then �� is a convex function if andonly if � is SPSD. Furthermore, �� is a strictlyconvex function if and only if � is SPD.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 38
Further Concepts forNonlinear Optimization
Examples of Convex Functions
Norm Function
Illustration of a norm function.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 39
Further Concepts forNonlinear Optimization
Examples of Convex Functions
The Gradient Inequality
The gradient inequality for convex functions:
Proposition 11 If �� is a differentiable convexfunction, then for any �� �,
�� � �� ������ � �� �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 40
Further Concepts forNonlinear Optimization
Examples of Convex Functions
The Subgradient Inequality
Even when a convex function is not differentiable, itmust satisfy the following “subgradient inequality.”
Proposition 12 If �� is a convex function, then forevery �, there must exist some vector � for which
�� � �� � ��� � �� for any � �
The vector � in this proposition is called a subgradientof �� at the point �.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 41
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem
We are given:
� points ��� � � � � �� � �� that have property “P”
� points ��� � � � � �� � �� that do not have property “P”
Illustration of the pattern classification problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 42
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem
We seek a vector % and a scalar � for which:
�%��� � � for all � � � � � � &
�%��� � � for all � � � � � �
We will then use %� � to predict whether or not anyother point � has property P or not.
� If %�� � �, then we declare that � has property P.
� If %�� � �, then we declare that � does not haveproperty P.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 43
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem%� � define the hyperplane
"� � � ��%�� ��
for which:
�%��� � � for all � � � � � � &
�%��� � � for all � � � � � � We would like the hyperplane"� � to be as far awayfrom the points ��� � � � � ��� ��� � � � � �� as possible.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 44
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem
The distance of the hyperplane ��� to any point �� is
���� ��
The distance of the hyperplane ��� to any point �� is
� ����
�If we normalize the vector � so that � � �, then the minimimdistance of the hyperplane ��� to the points ��� � � � � ��
��� � � � � � is:
�������� �� � � � � ��� ��� ����� � � � � � �����
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 45
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem
We would also like � and � to satisfy:
� � �, and
�������� �� � � � � ��� ��� ����� � � � � � ����� ismaximized.
��� � �����������Æ Æ
����
���� � � � � � �� � � � � �
� ���� � � � � �� � � � ��
� � ��
� � ��� � � �c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 46
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem� � � ��������� � Æ Æ
����
%��� � � � � � � � � � � &
� � %��� � � � � � � � �
% �
% � ��� � � �
� � is not a convex optimization problem,due to the presence of the constraint “ % .”
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 47
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem
��� � ���������Æ Æ
�� � ���� � � � Æ � � � � � � �
� � ���� � Æ � � � � � � �
��� � �
� � �� � � �
Transformation of variables:� %
Æ
� � �
Æ
��� Æ ���� %
� ����
� ���� �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 48
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem
��� � ���������Æ Æ
s.t. ���� � � � Æ � � � � � � �
� � ���� � Æ � � � � � � �
��� � �
� � �� � � �
� ��
Æ
� ��
Æ
���� � ����������� �
����
���� � � �� � � �� � � � � �
� ���� � �� � � �� � � � ��
� � ��� � � �c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 49
More Examples of ConvexOptimization Problems
A Pattern Classification Training Problem
���� � ������� ���
�� �
���� �� � � � � � � � � �
�� ���� � � � � � � � � �
� � �� � � �
� � is a convex problem.
Solve � � for ���, and substitute
% �
� �
� �
� �
Æ
�
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 50
More Examples of ConvexOptimization Problems
The Minimum Norm Problem
Given a vector �, we would like to find the closest pointto � that also satisfies the linear inequalities !� � �.
�� � ��������� �� �
����
!� � �
� � ��
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 51
More Examples of ConvexOptimization Problems
The Minimum Norm Problem
Illustration of the minimum norm problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 52
More Examples of ConvexOptimization Problems
The Fermat-Weber Problem
Given points ��� � � � � �� � ��, determine thelocation of a distribution center at the point � � �� thatminimizes the sum of the distances from � to each ofthe points ��� � � � � �� � ��.
'(� � �����������
��� �� ��
����
� � ��
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 53
More Examples of ConvexOptimization Problems
The Fermat-Weber Problem
Illustration of the Fermat-Weber problem.
'(� is a convex unconstrained problem.c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 54
More Examples of ConvexOptimization Problems
The Ball Circumscription Problem
Given points ��� � � � � �� � ��, determine thelocation of a distribution center at the point � � �� thatminimizes the maximum distance from � to any of thepoints ��� � � � � �� � ��.
� � � ��������� Æ Æ
����
�� �� � � � � � � � � �
� � ��
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 55
More Examples of ConvexOptimization Problems
The Ball Circumscription Problem
Illustration of the ball circumscription problem.
This is a convex (constrained) optimization problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 56
More Examples of ConvexOptimization Problems
The Analytic Center Problem
Given a system of linear inequalities !� � �, wewould like to determine a “nicely” interior point �� thatsatisfies !�� � �.
! � � �����������
�����!���
����
!� � ��
� � ��c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 57
More Examples of ConvexOptimization Problems
The Analytic Center Problem
Illustration of the analytic center problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 58
More Examples of ConvexOptimization Problems
The Analytic Center Problem
��� � ������
�����������
�� �
�� � �
� � ��
Proposition 13 Suppose that �� solves the analyticcenter problem ! � . Then for each �, we have:
��!���� ����
where
��� � ���
������!��� �c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 59
More Examples of ConvexOptimization Problems
The Analytic Center Problem
��� � ������
�����������
�� �
�� � �
� � ��
��� is not a convex problem, but it is equivalent to:
���� � �����������
��� ����������
����
�� � ��
� � ��
Now notice that ���� is a convex problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 60
More Examples of ConvexOptimization Problems
The Circumscribed Ellipsoid Problem
Given � points ��� � � � � �� � ��, determine an ellipsoid of
minimum volume that contains each of the points
��� � � � � �� � ��.
Illustration of the circumscribed ellipsoid problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 61
More Examples of ConvexOptimization Problems
The Circumscribed Ellipsoid Problem
Recall that an SPD matrix � and a given point � can be used todefine an ellipsoid in ��:
��� �� �� � �� ������ �� � ���
The volume of ��� is proportional to ���������
�.
Illustration of an ellipsoid.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 62
More Examples of ConvexOptimization Problems
The Circumscribed Ellipsoid Problem
Our problem is:� �� � �������� ���#����
�
#�)���� �� � *� �� � � � � � � &�
# is SPD.
� �� � �������� � �����#��
#�)���� �� � )��#�� � )� � � � � � � � � &
# is SPD.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 63
More Examples of ConvexOptimization Problems
The Circumscribed Ellipsoid Problem
���� � ���� � ����� ����
� �
�� � ��� � ������� � �� � � � � � � � � �
� is SPD.
Factor # �� where � is SPD
� �� � �������� � ���������
��)���� �� � )������� � )� � � � � � � � � &�
� is SPD.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 64
More Examples of ConvexOptimization Problems
The Circumscribed Ellipsoid Problem
���� � ���� � ����� �����
� �
�� � ��� � ��������� � �� � � � � � � � � �
� is SPD.
� �� � �������� �� ��������
��)
���� ��� � )� � � � � � � � � &�
� is SPD.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 65
More Examples of ConvexOptimization Problems
The Circumscribed Ellipsoid Problem
���� � ���� �� ����� ����
� �
�� � ����� � ��� � � � � � � � � �
� is SPD.
Next substitute � ��� to obtain:
���� � �������� ����������
���
���� ��� � � �� � � �� � � � � ��
� is SPD.– This is convex problem in the variables ���.
– We can recover � and � after solving ���� by substituting
� ��� and � �����.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 66
Classification of Nonlinear Optimization Problems
General Nonlinear Problem��� or ��� ��
���� ���� � �� � � � � � � �
Unconstrained Optimization
��� or ��� ��
���� � � ��c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 67
Classification of Nonlinear Optimization Problems
Linearly Constrained Problems
��� or ��� ��
���� !� � �
Linear Optimization��� ���
���� !� � �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 68
Classification of Nonlinear Optimization Problems
Quadratic Problem
��� ���� ������
���� !� � �
The objective function here is
�����
���� �
����
����������.
Quadratically Constrained Quadratic Problem
��� ���� ������
���� ��� ���
������ � �� � � � � � � �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 69
Classification of Nonlinear Optimization Problems
Separable Problem
���
����� ����
����
������������ � �� � � � �� � � � ��
Example of a separable problem
��� ��� � ��
� � ��
���� ��� ��� !"�� � #
������
� �� ������ � � �$
� � � � � � � � c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 70
Classification of Nonlinear Optimization Problems
Convex Problem
��� ���
���� ����� � �� � � � �� � � � ��
�� � �
where ��� is a convex function and ������ � � � � ����� areconvex functions, or:
��� ���
���� ����� � �� � � � �� � � � ��
�� � �
where ��� is a concave function and ������ � � � � ����� are
convex functions.c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 71
Solving Separable Convex Optimization via Linear Optimization
Consider the problem
��� ��� ��� �� � #��
���� ���� ���%�� � &
&���%��� �� � $
��� ��� �� �
This problem has:
1. linear constraints
2. a separable objective function � � ����� � ����� � #��
3. a convex objective function (since ���� and ���� are convexfunctions)
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 72
Solving Separable Convex Optimization via Linear Optimization
��� �� � ���� �������
���� ���� ���� ��� � �
���� ���� �� � �
��� ��� �� � �
We can approximate ��� and ��� bypiecewise-linear (PL) functions.
Suppose we know that � � �� � � and � � �� � �.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 73
Solving Separable Convex Optimization via Linear Optimization
PL approximation of ����.
����� � ��� �� � ' ����%���� �����
�� � ���� ���� ���
� ��� � �� � ��� � �� � ��� � � �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 74
Solving Separable Convex Optimization via Linear Optimization
PL approximation of ����.
����� � �� � � � ������ �$������ �'#�����
�� � ���� ���� ���
� ��� � �� � ��� � �� � ��� � �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 75
Solving Separable Convex Optimization via Linear Optimization
The linear optimization approximation then is:
��� �� ���� ����� ���� ��
����� ��
����� ���
����� ���
���� ���� ���� ��� � �
���� ���� �� � �
��� ��� �� � �
���� ���� ��� ��� ���� ���� ��� ��
� � ��� � �� � � ��� � �� � � ��� � �
� � ��� � �� � � ��� � �� � � ��� � �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 76
On the Geometry ofNonlinear Optimization
Extreme Point Solution
First example of the geometry of the solutionof a nonlinear optimization problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 77
On the Geometry ofNonlinear Optimization
Non-Extreme Point Boundary Solution
Second example of the geometry of the solutionof a nonlinear optimization problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 78
On the Geometry ofNonlinear Optimization
Interior Solution
Third example of the geometry of the solutionof a nonlinear optimization problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 79
On the Geometry ofNonlinear Optimization
Discussion
Unlike linear optimization, the optimal solution of anonlinear optimization problem need not be a “cornerpoint” of .
The optimal solution may be on the boundary or evenin the interior of .
The optimal solution will be characterized by how thecontours of �� are “aligned” with the feasible region
.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 80
Optimality Conditions forNonlinear Optimization
Convex Problem
Consider the convex problem:
� � � ��� ��
���� ���� � �� � � � � � � � �
��� ���� are convex functions.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 81
Optimality Conditions forNonlinear Optimization
The KKT Theorem
Convex Optimization
Theorem 2 (Karush-Kuhn-Tucker Theorem)Suppose that ��� ����� � � � � ���� are all convexfunctions. Then under very mild conditions, �� solves(CP) if and only if there exists ��� � �� � � � � � � ,such that
(�) ���� ���
������������ � (gradients line up)
(��) ������ �� � � (feasibility)
(���) ��� � � (multipliers must be nonnegative)
(��) ����� � ������ �� (multipliers for nonbinding constraints are zero)
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 82
Optimality Conditions forNonlinear Optimization
The KKT Theorem
Linear Optimization & Strong Duality...
Consider the following linear optimization problem andits dual:
! � � ��� ��� !"� � �����
��������
��� � � �� � � � � � � � ���
������� �
� � �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 83
Optimality Conditions forNonlinear Optimization
The KKT Theorem
...Linear Optimization & Strong Duality...
Let us look at the KKT Theorem applied here. If ��
solves LP, then there exists ���� � � � � � � for which:
(�) ���� ���
������������ � ��
���������� � (gradients line up)
(��) ������ �� � ��� �� �� � � ( primal feasibility)
(���) ��� � �
(��) ����� � ������ � ����� � ��
� �� �� (complementary slackness)
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 84
Optimality Conditions forNonlinear Optimization
The KKT Theorem
...Linear Optimization & Strong Duality
(�) ������ ��
������������� �� ��
��������� � � (gradients line up)
(��) ������� �� �� ��� �� �� � � ( primal feasibility)
(���) ��� � �
(��) ������ � ������� �� ������ � ��� �� � �� (complementary slackness)
– (��) is primal feasbility of ��– (�) and (���) together are dual feasibility of ��
– (��) is complementary slackness
The KKT Theorem states that �� and �� together must be
primal feasible and dual feasible, and must satisfy complementary
slackness.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 85
Optimality Conditions forNonlinear Optimization
The KKT Theorem
Geometry
g1(x) = 0
g2(x) = 0
g3(x) = 0
∇g1(x)
∇g2(x)
−∇f(x)
∇f(x) x–
Illustration of the KKT Theorem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 86
Optimality Conditions forNonlinear Optimization
The KKT Theorem
An example...
Consider the problem:
��� ���� � �� �'��� � �%��
���� ���
���� %�� � %
���
�����
�
��� ��� ����
� �&
��� � ���� � ���'��� � �%��
����� � ���
� ��� %�� %
����� � ���
�����
����� � ��� ���� ���
�&
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 87
Optimality Conditions forNonlinear Optimization
The KKT Theorem
...An example...
�� ���� � ���� ����� � ������
�� � ���
���� � ��� � ��
���
�����
� ���
��� � ��� ����
� ��
� ��� ��
�� ��� � �
$��� � �%��
�
������ ��
� ��
��� %��
� ������ ��
� ��
����
� ������ ��
� ��� ��
��
��
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 88
Optimality Conditions forNonlinear Optimization
The KKT Theorem
...An example...
��� ��� � ��� ���� � �����
���� ��� ��� � ��� � ��
��� ����� � ���
�� � ��� ���� � ��
Is �� ���� ���� ���� an optimal solution?
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 89
Optimality Conditions forNonlinear Optimization
The KKT Theorem
...An example...
�� ���� � ���� ����� � ������
�� � ���
���� � ��� � ��
���
�����
� ���
��� � ��� ����
� ��
We first check for feasibility of �� ����:����� �
����� ��� � �
����� �
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 90
Optimality Conditions forNonlinear Optimization
The KKT Theorem
...An example...
�� ���� � ���� ����� � ������
�� � ���
���� � ��� � ��
���
�����
� ���
��� � ��� ����
� ��
To check for optimality, we compute all gradients at ��:
��� �
�������
�����
�� �
�
����� �
� ���
�����
�� �
�
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 91
Optimality Conditions forNonlinear Optimization
The KKT Theorem
...An example
�� ���� � ���� ����� � ������
�� � ���
���� � ��� � ��
���
�����
� ���
��� � ��� ����
� ��
We next try to solve for �� � � �� � � �� � in the system:
����
% �
���
��'
�
�����
��'
���
�����
�
� �
��� ��
�
�
�
�� � ����� ���� ���� � � � �'� solves this system in nonnegativevalues, and �� � .
Therefore �� is an optimal solution to this problem.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 92
Optimality Conditions forNonlinear Optimization
KKT Theorem with Different Formats of Constraints
Convex Optimization with Linear Equations
����� � �� ����
�� � ����� � �� � � �
��� � � �� � � �
Theorem 3 (Karush-Kuhn-Tucker Theorem) Suppose that ���� ����� are allconvex functions for � � �. Then under very mild conditions, �� solves (CPE) ifand only if there exists ��� � � � � � and ��� � � � such that
(�) ������ ��
������������ �
�����
����� � � (gradients line up)
(����) ������� �� � � � � � (feasibility)
(����) ��� �� �� � � � � � (feasibility)
(���) ��� � � � � � (nonnegative multipliers on inequalities)
(��) ������ � ������� � � � � �� (complementary slackness)
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 93
Optimality Conditions forNonlinear Optimization
KKT Theorem with Different Formats of Constraints
Non-convex Optimization
����� � �� ����
�� � ����� � �� � � �
����� � �� � � �
Theorem 4 (Karush-Kuhn-Tucker Theorem) Under some very mild conditions,if �� solves (NCE), then there exists ��� � � � � � and ��� � � � such that
(�) ������ ��
������������ �
����
��������� � � (gradients line up)
(����) ������� �� � � � � � (feasibility)
(����) ������ �� � � � � � (feasibility)
(��� ) ��� � � (nonnegative multipliers on inequalities)
(��) ������ � ������� � �� (complementary slackness)
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 94
A Few Concluding Remarks
Nonconvex optimization problems can be difficult tosolve.
This is because local optima may not be global optima.
Most algorithms are based on calculus, and so canonly find a local optimum.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 95
A Few Concluding Remarks
There are a large number of solution methods forsolving nonlinear constrained problems.
Today, the most powerful methods fall into two classes:
�methods that try to generalize the simplex algorithmto the nonlinear case.
�methods that generalize the barrier method to thenonlinear case.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 96
A Few Concluding Remarks
Quadratic Problems. A quadratic problem is “almostlinear”, and can be solved by a special implementationof the simplex algorithm, called the complementarypivoting algorithm.
Quadratic problems are roughly eight times harder tosolve than linear optimization problems of comparablesize.
c�2002 Massachusetts Institute of Technology. All rights reserved. 15.094 97