EE8950TomLuo

Lecture6:Duality

•Lagrangedualfunction

•Lagrangedualproblem

•strongdualityandSlater’scondition

•KKToptimalityconditions

•sensitivityanalysis

•equalityconstraints

•generalizedinequalities

•theoremsofalternatives

1

EE8950TomLuo

Lagrangian

standardformproblem(withoutequalityconstraints)

minimizef0(x)

subjecttofi(x)≤0,i=1,...,m

•optimalvaluep?,domainD

•calledprimalproblem(incontextofduality)

(fornow)wedon’tassumeconvexity

LagrangianL:Rn+m

→R

L(x,λ)=f0(x)+λ1f1(x)+···+λmfm(x)

•λicalledLagrangemultipliersordualvariables

•objectiveisaugmentedwithweightedsumofconstraintfunctions

2

EE8950TomLuo

Lagrangedualfunction

(Lagrange)dualfunctiong:Rm

→R∪{−∞}

g(λ)=infx

L(x,λ)=infx

(f0(x)+λ1f1(x)+···+λmfm(x))

•minimumofaugmentedcostasfunctionofweights

•canbe−∞forsomeλ

•gisconcave(eveniffinotconvex!)

example:LPminimizec

Tx

subjecttoaTix−bi≤0,i=1,...,m

NotethatL(x,λ)=cTx+

m∑

i=1

λi(aTix−bi)=−b

Tλ+(A

Tλ+c)

Tx

henceg(λ)=

{

−bTλifA

Tλ+c=0

−∞otherwise

3

EE8950TomLuo

Lowerboundproperty

ifλ�0andxisprimalfeasible,then

g(λ)≤f0(x)

proof:iffi(x)≤0andλi≥0,

f0(x)≥f0(x)+∑

i

λifi(x)≥infz

(

f0(z)+∑

i

λifi(z)

)

=g(λ)

f0(x)−g(λ)iscalledthedualitygapof(primalfeasible)xandλ�0

minimizeoverprimalfeasiblextoget,foranyλ�0,

g(λ)≤p?

λ∈Rm

isdualfeasibleifλ�0andg(λ)>−∞

dualfeasiblepointsyieldlowerboundsonoptimalvalue!

4

EE8950TomLuo

Lagrangedualproblem

let’sfindbestlowerboundonp?:

maximizeg(λ)

subjecttoλ�0

•called(Lagrange)dualproblem

(associatedwithprimalproblem)

•alwaysaconvexproblem,evenifprimalisn’t!

•optimalvaluedenotedd?

•wealwayshaved?≤p

?(calledweakduality)

•p?−d

?isoptimaldualitygap

5

EE8950TomLuo

Strongduality

forconvexproblems,we(usually)havestrongduality:

d?=p

?

whenstrongdualityholds,dualoptimalλ?

servesascertificateofoptimalityforprimal

optimalpointx?

manyconditionsorconstraintqualificationsguaranteestrongdualityforconvexproblems

Slater’scondition:ifprimalproblemisstrictlyfeasible(andconvex),i.e.,thereexists

x∈relintDwith

fi(x)<0,i=1,...,m

thenwehavep?=d

?

6

EE8950TomLuo

Dualoflinearprogram

(primal)LPminimizec

Tx

subjecttoAx�b

•nvariables,minequalityconstraints

dualofLPis(aftermakingimplicitequalityconstraintsexplicit)

maximize−bTλ

subjecttoATλ+c=0

λ�0

•dualofLPisalsoanLP(indeed,instdLPformat)

•mvariables,nequalityconstraints,mnonnegativitycontraints

forLPwehavestrongdualityexceptinone(pathological)case:primalanddualboth

infeasible(p?=+∞,d

?=−∞)

7

EE8950TomLuo

Dualofquadraticprogram

(primal)QPminimizex

TPx

subjecttoAx�b

weassumeP�0forsimplicityLagrangianisL(x,λ)=xTPx+λ

T(Ax−b)

∇xL(x,λ)=0yieldsx=−(1/2)P−1

ATλ,hencedualfunctionis

g(λ)=−(1/4)λTAP

−1A

Tλ−b

Tλ

•concavequadraticfunction

•allλ�0aredualfeasible

dualofQPismaximize−(1/4)λ

TAP

−1A

Tλ−b

Tλ

subjecttoλ�0

...anotherQP

8

EE8950TomLuo

Dualityinalgorithms

manyalgorithmsproduceatiterationk

•aprimalfeasiblex(k)

•andadualfeasibleλ(k)

withf0(x(k)

)−g(λ(k)

)→0ask→∞

henceatiterationkweknowp?∈[

g(λ(k)

),f0(x(k)

)]

•usefulforstoppingcriteria

•algorithmsthatusedualsolutionareoftenmoreefficient(e.g.,LP)

9

EE8950TomLuo

Nonheuristicstoppingcriteria

absoluteerror=f0(x(k)

)−p?≤ε

stoppingcriterion:until(

f0(x(k)

)−g(λ(k)

)≤ε)

relativeerror=f0(x

(k))−p

?

|p?|≤ε

stoppingcriterion:

until

(

g(λ(k)

)>0&f0(x(k))−g(λ(k))

g(λ(k))≤ε

)

or

(

f0(x(k)

)<0&f0(x(k))−g(λ(k))

−f0(x(k))≤ε

)

achievetargetvalue`or,prove`isunachievable

(i.e.,determineeitherp?≤`orp

?>`)

stoppingcriterion:until(

f0(x(k)

)≤`org(λ(k)

)>`)

10

EE8950TomLuo

Complementaryslackness

supposex?,λ

?areprimal,dualfeasiblewithzerodualitygap(hence,theyareprimal,dual

optimal)

f0(x?)=g(λ

?)=inf

x

(

f0(x)+

m∑

i=1

λ?ifi(x)

)

≤f0(x?)+

m∑

i=1

λ?ifi(x

?)

hencewehave∑

mi=1λ

?ifi(x

?)=0,andso

λ?ifi(x

?)=0,i=1,...,m

•calledcomplementaryslacknesscondition

•ithconstraintinactiveatoptimum=⇒λi=0

•λ?i>0atoptimum=⇒ithconstraintactiveatoptimum

11

EE8950TomLuo

KKToptimalityconditions

suppose

•fiaredifferentiable

•x?,λ

?are(primal,dual)optimal,withzerodualitygap

bycomplementaryslacknesswehave

f0(x?)+

∑

i

λ?ifi(x

?)=inf

x

(

f0(x)+∑

i

λ?ifi(x)

)

i.e.,x?

minimizesL(x,λ?)

therefore

∇f0(x?)+

∑

i

λ?i∇fi(x

?)=0

12

EE8950TomLuo

soifx?,λ

?are(primal,dual)optimal,withzerodualitygap,theysatisfy

fi(x?)≤0

λ?i≥0

λ?ifi(x

?)=0

∇f0(x?)+

∑

iλ?i∇fi(x

?)=0

theKarush-Kuhn-Tucker(KKT)optimalityconditions

conversely,iftheproblemisconvexandx?,λ

?satisfyKKT,thentheyare(primal,dual)

optimal

13

EE8950TomLuo

Geometricinterpretationofduality

considerset

A={(u,t)∈Rm+1

|∃xfi(x)≤ui,f0(x)≤t}

•Aisconvexiffiare

•forλ�0,

g(λ)=inf

{

[

λ

1

]

T[

u

t

]

∣

∣

∣

∣

∣

[

u

t

]

∈A

}

PSfragreplacements

u

t

A

t+λT

u=g(λ)

g(λ)[

λ1

]

14

EE8950TomLuo

(Ideaof)proofofSlater’stheoremproblemconvex,strictlyfeasible=⇒strongduality

PSfragreplacements

u

t

A

[

1

λ?

]

p?

•(0,p?)∈∂A⇒∃supportinghyperplaneat(0,p

?):

(u,t)∈A=⇒µ0(t−p?)+µ

Tu≥0

•µ0≥0,µ�0,(µ,µ0)6=0

•strongduality⇔∃supportinghyperplanewithµ0>0:forλ?=µ/µ0,wehave

p?≤t+λ

?Tu∀(t,u)∈A,p

?≤g(λ

?)

•Slater’scondition:thereexists(u,t)∈Awithu≺0;impliesthatallsupporting

hyperplanesat(0,p?)arenon-vertical(µ0>0)

15

EE8950TomLuo

Sensitivityanalysisviaduality

definep?(u)astheoptimalvalueof

minimizef0(x),subjecttofi(x)≤ui,i=1,...,m

0

0

PSfragreplacements

u

p?(u

)

epip?

p?(0)−λ?Tu

λ?

giveslowerboundonp?(u):p

?(u)≥p

?−∑

mi=1λ

?iui

•ifλ?ilarge:ui<0greatlyincreasesp

?

•ifλ?ismall:ui>0doesnotdecreasep

?toomuch

ifp?(u)isdifferentiable,λ

?i=−

∂p?(0)

∂ui

,λ?iissensitivityofp

?w.r.t.ithconstraint

16

EE8950TomLuo

Equalityconstraints

minimizef0(x)

subjecttofi(x)≤0,i=1,...,m

hi(x)=0,i=1,...,p

•optimalvaluep?

•againassume(fornow)notnecessarilyconvex

defineLagrangianL:Rn+m+p

→Ras

L(x,λ,ν)=f0(x)+

m∑

i=1

λifi(x)+

p∑

i=1

νihi(x)

dualfunctionisg(λ,ν)=infxL(x,λ,ν)

(λ,ν)isdualfeasibleifλ�0andg(λ,ν)>−∞

(nosignconditiononν)

17

EE8950TomLuo

lowerboundproperty:ifxisprimalfeasibleand(λ,ν)isdualfeasible,theng(λ,ν)≤

f0(x),hence

g(λ,ν)≤p?

dualproblem:findbestlowerbound

maximizeg(λ,ν)

subjecttoλ�0

(noteνunconstrained),optimalvalued?

weakduality:d?≤p

?always

strongduality:ifprimalisconvexthen(usually)d?=p

?

Slatercondition:ifprimalisconvex(i.e.,ficonvex,hiaffine)andstrictlyfeasible,i.e.,

thereexistsx∈relintDs.t.

fi(x)<0,hi(x)=0,

thend?=p

?

18

EE8950TomLuo

Example:equalityconstrainedleast-squares

minimizexTx

subjecttoAx=b

Aisfat,fullrank(solutionisx?=A

T(AA

T)−1

b)

dualfunctionis

g(ν)=infx

(

xTx+ν

T(Ax−b)

)

=−1

4ν

TAA

Tν−b

Tν

dualproblemis

maximize−14ν

TAA

Tν−b

Tν

solution:ν?=−2(AA

T)−1

b

cancheckd?=p

?

19

EE8950TomLuo

KKToptimalityconditions

assumefi,hidifferentiable

ifx?,λ

?,ν

?areoptimal,withzerodualitygap,thentheysatisfyKKTconditions

fi(x?)≤0,hi(x

?)=0

λ?i≥0

λ?ifi(x

?)=0

∇f0(x?)+

∑

iλ?i∇fi(x

?)+

∑

iν?i∇hi(x

?)=0

conversely,iftheysatisfyKKTandtheproblemisconvex,thenx?,λ

?,ν

?areoptimal

example:optimalityconditionsforequalityconstraintsonly

minimizef0(x)

subjecttoAx=b

x?

optimal⇐⇒if∃ν?

s.t.∇f0(x?)+A

Tν

?=0

20

EE8950TomLuo

Introducingequalityconstraints

idea:simpletransformationofprimalproblemcanleadtoverydifferentdual

example:unconstrainedgeometricprogramming

primalproblem:

minimizelog

m∑

i=1

exp(aTix−bi)

dualfunctionisconstantg=p?

(wehavestrongduality,butit’suseless)

nowrewriteprimalproblemas

minimizelog

m∑

i=1

expyi

subjecttoy=Ax−b

21

EE8950TomLuo

letusintroduce

•mnewvariablesy1,...,ym

•mnewequalityconstraintsy=Ax−b

dualfunction

g(ν)=infx,y

(

log

m∑

i=1

expyi+νT(Ax−b−y)

)

•infimumis−∞ifATν6=0

•assumingATν=0,let’sminimizeovery:

eyi

∑

nj=1e

yj=νi

solvableiffνi>0,1Tν=1

g(ν)=−∑

i

νilogνi−bTν

22

EE8950TomLuo

•sameexpressionifν�0,1Tν=1(0log0=0)

dualproblem

maximize−bTν−

∑

i

νilogνi

subjectto1Tν=1,(ν�0)

ATν=0

moral:trivialreformulationcanyielddifferentdual

23

EE8950TomLuo

Generalizedinequalities

minimizef0(x)

subjecttofi(x)�Ki0,i=1,...,L

•�KiaregeneralizedinequalitiesonRmi

•fi:Rn→R

miareKi-convex

LagrangianL:Rn×R

m1×···×RmL→R,

L(x,λ1,...,λL)=f0(x)+λT1f1(x)+···+λ

TLfL(x)

dualfunction

g(λ1,...,λL)=infx

(

f0(x)+λT1f1(x)+···+λ

TLfL(x)

)

λidualfeasibleifλi�K?i

0,g(λ1,...,λL)>−∞

24

EE8950TomLuo

lowerboundproperty:ifxprimalfeasibleand

(λ1,...,λL)isdualfeasible,then

g(λ1,...,λL)≤f0(x)

(hence,g(λ1,...,λL)≤p?)

dualproblemmaximizeg(λ1,...,λL)

subjecttoλi�K?i

0,i=1,...,L

weakduality:d?≤p

?always

strongduality:d?=p

?usually

Slatercondition:ifprimalisstrictlyfeasible,i.e.,

∃x∈relintD:fi(x)≺Ki0,i=1,...,L

thend?=p

?

25

EE8950TomLuo

Example:semidefiniteprogramming

minimizecTx

subjecttoF0+x1F1+···+xnFn�0

Lagrangian(multiplierZ=ZT∈R

m×m)

L(x,Z)=cTx+TrZ(F0+x1F1+···+xnFn)

dualfunction

g(Z)=infx

(

cTx+TrZ(F0+x1F1+···+xnFn)

)

=

{

TrF0ZifTrFiZ+ci=0,i=1,...,n

−∞otherwise

dualproblemmaximizeTrF0Z

subjecttoTrFiZ+ci=0,i=1,...,n

Z=ZT�0

strongdualityholdsifthereexistsxwithF0+x1F1+···+xnFn≺0

26

EE8950TomLuo

Theoremofalternatives

f1,...,fmconvexwithdomfi=Rn

exactlyoneofthefollowingistrue:

1.thereexistsxwithfi(x)<0,i=1,...,m

2.thereexistsλ6=0withλ�0,

g(λ)=infx

(λ1f1(x)+···+λmfm(x))≥0

•calledalternatives

•useinpractice:λthatsatisfies2ndconditionprovesfi(x)<0isinfeasible

example:fi(x)=aTix−bi

1.thereexistsxwithAx≺b

2.thereexistsλ�0,λ6=0,bTλ≤0,A

Tλ=0

27

EE8950TomLuo

proof.fromconvexduality:

primalproblemminimizet

subjecttofi(x)≤t,i=1,...,m

(variablesx,t)

dualproblemmaximizeg(λ)

subjecttoλ�0

1Tλ=1

•Slater’sconditionissatisfied,hencep?=d

?

•1stalternative:⇐⇒p?

<0

•2ndalternative:⇐⇒p?≥0

28