agenda - statweb.stanford.educandes/math301/lectures/conic... · agenda 1 duality for lp 2 theorem...
TRANSCRIPT
Agenda
1 Duality for LP2 Theorem of alternatives3 Conic Duality4 Dual cones5 Geometric view of cone programs6 Conic duality theorem7 Examples
Lower bounds on LPs
By eliminating variables (if needed) any LP can be put in inequality form
minimize c
T
x
subject to Ax � b
We use this form for simplicity
Purpose of Lagrange duality: get a lower bound on optimal value p
?
Lagrange duality
Lagrangian: L : Rn ⇥ Rm ! R
L(x,�) = c
T
x� �
T (Ax� b) �: Lagrange multipliers
(Lagrange) dual function
g(�) = infx2Rn L(x,�) =
(b
T
� A
T
� = c
�1 otherwise
Suppose � � 0 & x is feasible
L(x,�) = c
T
x� �
T (Ax� b) c
T
x
This implies that g(�) c
T
x for all feasible x
) g(�) infx:Ax�b
c
T
x = p
?
! Dual function always gives a lower bound on optimal value p
?
Dual problem
Dual problem: best lower bound on p
?
maximize g(�)subject to � � 0
()maximize b
T
�
subject to A
T
� = c
� � 0
(Lagrange) dual problem
This is an LP
Optimal value d
? obeys d? p
?: weak duality
LP duality
Theorem
For LP, strong duality always holds (remarkable and very useful fact)
d
? = p
?
p
? = 1 (primal infeasible) =) d
? = p
?
p
? = �1 =) d
? = �1 (dual infeasible)
Theorem of alternatives and strong LP duality
min {cTx : Ax � b}
How do we decide if LP is feasible?
How can we certify feasible set is empty?
One possible approach: combine inequalities and get a contradiction
� � 0 and Ax � b =) �
T
Ax � �
T
b
Get such a contradiction if AT
� = 0 and b
T
� > 0 (only way to be sure)
Remarkably, this is su�cient but also necessary
Theorem (Theorem of alternatives)
Exactly one of (1) and (2) is true
(1) 9x s.t. Ax � b
(2) 9� � 0 s.t. AT
� = 0 and b
T
� > 0
Lower bounds for LPs
min {cTx : Ax � b}
a p
? , {x : Ax � b and c
T
x < a} = ;
Theorem of alternatives: a p
? i↵
9(�0,�) � 0 s .t.��0c+A
T
� = 0��0 + b
T
� � 0
If LP is feasible, can take �0 > 0, hence a p
? i↵ system below has a solution
(D)� � 0A
T
� = c
b
T
� � a
Proposition
(D) has a sol (�, a) =) a p
?
LP is feasible and a p
? =) existence of a sol to (D)
Strong duality
Proposition
(D) has a sol (�, a) =) a p
?
LP is feasible and a p
? =) existence of a sol to (D)
(�, a) sol to (D) =) (�, bT�) sol to (D)
Since b
T
� � a, best lower bound
max {bT� : AT
� = c,� � 0}
ConclusionThis proves strong duality
� feasible for dual problem =) b
T
� p
? (d? p
?)
If LP feasible, then for every a p
?, 9 dual feasible � with b
T
� � a
) d
? = p
?
Proof of theorem of alternatives
Similar notion for cone programs?
Conic programminimize c
T
x
subject to Ax ⌫K
b
Interested in something similar to LP’s
� � 0 and Ax� b � 0 =) h�, Ax� bi � 0
For cones, we would like a set C such that
� 2 C and a ⌫K
0 =) h�, ai � 0
If � 2 C and x is feasible, we get a lower bound
hc, xi � h�, Ax� bi hc, xi
Or similarly
min {cTx : Ax ⌫K
0}
For LP’s: � � 0 gives
Ax � b =) �
T
Ax � �
T
b
If AT
� = c, we get a lower bound
For cones, we would like a set C such that
� 2 C and a ⌫K
0 =) h�, ai � 0
Best lower boundmax {bT� : � 2 C and A
T
� = c}
Dual cones
DefinitionDual cone K
⇤
K
⇤ = {� 2 Rn : �T
a � 0 8a 2 K}
Properties of dual cone
K 6= ; (nonempty set)
1K
⇤ is a closed convex cone2 int(K) 6= ; =) K
⇤ is pointed3
K closed, convex and pointed =) int(K⇤) 6= ;
4K closed, convex and pointed =) K
⇤ closed and convex(K⇤)⇤ = K
Corollary: K proper () K
⇤ proper
Dual of a conic problem
Primal
(P)minimize c
T
x
subject to Ax ⌫K
b
LagrangianL(x,�) = c
T
x� �
T (Ax� b) � 2 K
⇤
Dual
(D)maximize b
T
�
subject to A
T
� = c
� 2 K
⇤
Weak duality: d⇤ p
⇤
x feasible, � feasible =) b
T
� c
T
x
Dual of conic problem, continued
(P)minimize c
T
x
subject to A
i
x ⌫Ki bi, i = 1, . . . ,m
(D)maximize
Pb
T
i
�
i
subject toP
A
T
i
�
i
= c
�
i
⌫K
⇤i0, i = 1, . . . ,m
Geometrical view of primal/dual pair
(P)minimize c
T
x
subject to Ax ⌫K
b
(D)maximize b
T
�
subject to A
T
� = c
� ⌫K
⇤ 0
Primal: minimize linear function over an a�ne slice of cone K
Dual: maximize linear function over an a�ne slice of cone K
⇤
Same problem: only di↵erence due to how we represent the problem
Assume 9d : c = A
T
d () c 2 row(A)otherwise dual is infeasible: d? = �1 and p
? 2 {±1}Assume for simplicity A has full column rank (so that c = A
T
d as a solution)
Primal feasible set: v = Ax� b
(i) v 2 V � b, V = range(A)
(ii) v ⌫K
0
(iii) Gives cTx = d
T
Ax = d
T
b+ d
T
v
Conic program can be formulated as
min {dT b+ d
T
v : v 2 V � b, v ⌫K
0}
Dual feasible set:
A
T
� = c,� 2 K
⇤ () � 2 V
? + d, � 2 K
⇤
So that dual problem is
max {bT� : � 2 V
? + d,� ⌫K
⇤ 0}
Geometric view of primal/dual pair, continued
K*
K
pri
ma
l fe
as
ible
R - b
dual feasible
Symmetry of duality
(P)minimize c
T
x
subject to Ax ⌫K
b
(D)maximize b
T
�
subject to A
T
� = c
� ⌫K
⇤ 0
If K is proper, then (P) is dual to (D) because K
⇤⇤ = K
dual of (D)minimize �⌫
T
c
subject to v ⌫K
0v = �A⌫ � b
() minimize c
T
x
subject to Ax� b ⌫K
0
Examples of dual cones
The following are all self-dual cones:
(a) K = Rn
+
K
⇤ = {� :X
�
i
a
i
� 0 8a � 0} = Rn
+
(b) K = L ⇢ Rn+1, L = {(x, t) : kxk t}
(y, s) 2 K
⇤ () (y, x) + st � 0 8(x, t) 2 L
t
x
L = {(x, t) : kxk t}
(y, s) 2 L
⇤ () hy, xi+ st � 0 8(x, t) 2 L
Claim: L⇤ = L
(i) L ⇢ L
⇤
Take (y, s) 2 L,hy, xi+ st � �kykkxk+ st � 0
(ii) L
⇤ ⇢ L
Take (y, s) 2 L
⇤ and set (x, t) = (�y, kyk) 2 L
�hy, yi+ skyk � 0 =) kyk s () (y, s) 2 L
(c) K = S
n
+
K
⇤ = {⇤ 2 S
n : Tr(⇤X) � 0 8X ⌫ 0}Claim: K⇤ = K
(i) K
⇤ ⇢ S
n+
Take ⇤ 2 K
⇤. Then for all x 2 Rn
Tr(⇤xxT ) = Tr(xT⇤x) = x
T⇤x � 0
and so ⇤ ⌫ 0
(ii) S
n+ ⇢ K
⇤
Take ⇤ ⌫ 0 and decompose X ⌫ 0 as X =P
�kxkxTk ,with �k � 0
Tr(X⇤) =X
k
�kTr(xkxTk ⇤) =
X
k
�kxTk ⇤xk � 0
Implication of conic self duality
Primal LP SOCP SDP
Dual LP SOCP SDP
Example: minimum total-variation denoising
minimize kxkTVsubject to kx� bk �
()minimize
Pt
ij
subject to kx� bk �
kDij
xk t
ij
Constraints
kx� bk � ()x� b
�
�2 L0 ()
I 00 0
� x
t
�⌫
L0
b
��
�
(Dij
x, t
ij
) 2 L
ij
()D 00 I
� x
t
�⌫
L
0 L =Y
L
ij
Dual
maximize �
T
0
b
��
�
subject to �0 ⌫L0 0 �
ij
⌫Lij 0
�I 00 0
� u0
s0
��D
T 00 I
� u
s
�+
01
�= 0
Dual variables
�0 =
u0
s0
�: ku0k s0 �
ij
=
u
ij
s
ij
�: ku
ij
k s
ij
Dual problem (SOCP)
max hu0, bi � s0�
s. t. ku0k s0, kuij
k s
ij
u0 +D
T
u = 0s = 1
() max �hb,DT
ui � �kDT
uks. t. ku
ij
k 1
Example: SDP
maximize hc, xisubject to x1F1 + . . .+ x
m
F
m
⌫ F0 (Fi
2 S
p)
Dual cone problem
maximize hF0,⇤isubject to ⇤ ⌫ 0
hFi
,⇤i = c
i
, i = 1, . . . ,m
minimize kA0 � (x1A1 + . . .+ x
m
A
m
)k
SDP formulation
minimize t
subject to
tI A(x)
A
T (x) tI
�⌫ 0
with A(x) = A0 � (x1A1 + . . .+ x
m
A
m
)
LMI () t
I 00 I
��X
i
x
i
0 A
i
A
T
i
0
�+
0 A0
A
T
0 0
�⌫ 0
Dual problem
maximize h�0 A0
A
T
0 0
�,⇤i
subject to ⇤ ⌫ 0
h0 A
i
A
T
i
0
�,⇤i = 0
hI 00 I
�,⇤i = 1
With ⇤ =
⇤11 ⇤12
⇤T
12 ⇤22
�, this gives
maximize �2Tr(AT
0 ⇤12)subject to Tr(AT
i
⇤12) = 0Tr(⇤) = 1⇤ ⌫ 0
()
maximize �2Tr(AT
0 ⇤12)subject to Tr(AT
i
⇤12) = 0Tr(⇤11) + Tr(⇤22) = 1⇤11 ⇤12
⇤T
12 ⇤22
�⌫ 0
Conic duality theorem
Can get results paralleling LP duality provided primal is strictly feasible
Strict feasibility means that V � b \ int(K) 6= ;
Theorem
(1) Assume (P) is strictly feasible (9x : Ax �K
b) and p
? 6= �1. Then (D) issolvable and d
? = p
?.
(2) Assume (D) is strictly feasible (9� : � �K
⇤ 0, AT
� = c) and d
? 6= 1. Then(P) is solvable and d
? = p
?.
(3) Assume one of the problems is bounded and strictly feasible. Then (x,�)primal/dual optimal i↵
b
T
� = c
T
x () �
T (Ax� b) = 0
Proof(1) Claim: If (P) is strictly feasible and 9x : Ax �
K
b
then (D) is solvable and d
? = p
?
Enough to show that 9� dual feasible s.t. bT� � p
?
If c = 0, then p
? = 0 and �
? = 0 does the job. So we assume c 6= 0.
M = {Ax� b : x 2 Rn
, c
T
x p
?}
M 6= ; (image of halfspace)
K
R - b
M d
M \ int(K) = ;Why? Assume v 2 M \ int(K) () v = Ax0 � b with Ax0 � b �
K
0 andc
T
x0 p
? and hence x0 is primal optimal9✏ ⌫ 0 s.t. Ax� b � 0 8x : kx� x0k ✏
c 6= 0 =) 9x s.t. kx� x0k ✏ and c
Tx < c
Tx0 p
?
=)(=
M is convex, nonempty, and M \ int(K) = ;
Separation theorem for convex sets: 9� 6= 0 s.t
supv2M
h�, vi infv2int(K)
h�, vi
hence �
T
v bounded below on int(K)
=) �
T
y � 0 8y 2 int(K)=) �
T
y � 0 8y 2 K
() � 2 K
⇤
In summary
supv2M
h�, vi 0 & � 2 K
⇤ () � 2 K
⇤
�
T (Ax� b) 0 8x : cTx p
?
! Linear form bounded above on a half-plane. This is possible only if
A
T
� = µc µ � 0
We claim µ > 0Why? µ = 0 =) A
T
� = 0, �
T
b � 0Take x0 s.t. Ax0 � b �
K
0
� ⌫K
⇤ 0 and � 6= 0 =) �
T (Ax0 � b) > 0 =) ��
T
b > 0
This is a contradiction
�
? = µ
�1� obeys
�
? 2 K
?
, A
T
�
? = c, hc, xi h�?
, bi 8x : cTx p
?
Taking an x such that cTx = p
? =) value of dual objective is at least p?
End of proof
(2) Same by symmetry
(3) x primal feasible, � dual feasible =) c
T
x� b
T
� = (Ax� b)T�
! Complementary slackness i↵ zero duality gap
Enough to show (x,�) optimal () DG = 0
DG is zero gives
c
T
x� b
T
� = c
T
x� p
? + d
? � b
T
�+ p
? � d
? = 0
since c
T
x� p
? = 0 and d
? � b
T
� = 0 must holdIn conclusion,
x primal optimal� dual optimal
nota bene: no strict feasibility required
Conversely, p? = d
? =) c
T
x = b
T
�
Example
(P) minimize c
T
x
subject to x1F1 + . . .+ x
n
F
n
⌫ F0
(D) maximize hF0,⇤isubject to ⇤ ⌫ 0
hFi
,⇤i = c
i
Primal / dual strict feasibility
(x,⇤) optimal () c
T
x� hF0,⇤i = hX
x
i
F
i
� F0,⇤i = 0
Example for which p? > d?
minimize x1
subject to
2
40 x1 0x1 x1 + x2 00 0 x1 + 1
3
5 ⌫ 0
Feasible set:0 x1
x1 x1 + x2
�⌫ 0
x1 + 1 � 0()
x1 + x2 � 0�x
21 � 0
x1 � �1() x1 = 0
x2 � 0
) p
? = 0
Dual 2
40 0 00 0 00 0 1
3
5+ x1
2
40 1 01 1 00 0 1
3
5+ x2
2
40 0 00 1 00 0 0
3
5 ⌫ 0
which givesmaximize ��33
subject to
2
4�11
1��332 ⇤
1��332 0 ⇤⇤ ⇤ �33
3
5 ⌫ 0
⇤ dual feasible =) �33 = 1) d
? = �1
SDP violates conditions of conic duality theorem
Both (P) and (D) are feasible, but not strictly
6= LP where strong duality always holds
Nuclear norm
TheoremOperator norm and nuclear norm are dual
Proof: want to show kXk⇤ = maxkY k1
hY,Xi
(a) Take SVD of X = U⌃V T and set Y = UV
T . Then
Tr(Y T
X) = Tr(⌃) = kXk⇤ =) maxkY k1
hY,Xi � kXk⇤
(b) In the other direction
(D)maximize hY,Xi
subject to
I Y
Y
T
I
�⌫ 0
duality$ (P)minimize 1
2Tr(⇤1) +12Tr(⇤2)
subject to
⇤1 X
X
T ⇤2
�⌫ 0
(D) has variable Y and (P) has variables ⇤1,⇤2
⇤1 = U⌃UT ⇤2 = V ⌃V T =)⇤1 X
X
T ⇤2
�=
U
V
� ⌃ 00 ⌃
� U
V
�T
val(D) val(P) Tr(⌃) = kXk⇤
Consequence
minimize kXk⇤subject to X 2 C
If C is SDP-representable, this is an SDP; e.g.A(X) = b
kA(X)� bk2 ✏
kA⇤(A(X)� b)k �
. . .
CVX knows about this