ellipsoid 4

8/2/2019 Ellipsoid 4

1/26

T he E llipsoid m ethod

R.M. Freund/ C. Roos (MIT/TUD)e-mail: [email protected]

URL: http://www.isa.ewi.tudelft.nl/roos

WI 4218March 21, A.D. 2007

Optimization Group 1/26


2/26

Outline

Sherman-Morrison formula

Ellipsoids

Ellipsoid method

Basic construction Prototypical iteration

Iteration bound

Two theorems Two more theorems

Optimization with the ellipsoid method

Convexity of the homogeneous problem The Key Proposition



3/26

In memory of George Danzig and Leonid Khachyan (1)

Leonid Khachiyan (1952-2005) passed away Friday April 29 at the age of 52. He died in hissleep, apparently of a heart attack, colleagues said. Khachiyan was best known for his 1979

use of the ellipsoid algorithm, originally developed for convex programming, to give the first

polynomial-time algorithm to solve linear programming problems. While the simplex algorithm

solved linear programs well in practice, Khachiyan gave the first formal proof of an efficientalgorithm in the worst case.

He was among the worlds most famous computer scientists, said Haym Hirsh, chairman of

the computer science department at Rutgers.

George B. Dantzig, the father of Linear Programming and the creator of the Simplex Method,

died at the age of 90 on May 13. In a statement INFORMS President Richard C. Larsonmourned his death (see http://www.informs.org/Press/dantzigobit.htm). The tributes in-

cluded obituaries in the Washington Post, the Francisco Chronicle, Mercury News and New

York Times. National Public Radio commentator Keith Devlin remembered George Dantzig

in a broadcast on Saturday, May 21.



4/26

In memory of George Danzig and Leonid Khachyan (2)



5/26

Leonid Khachyan in Shanghai (with Bai, Peng and Terlaky, 2002)



6/26

Sherman-Morrison formulaLet Q,R,S,T be matrices such that

Q and Q + RST are nonsingular.

R and S are n

k matrices of rank k

n.

Then (Q + RST)1 = Q1 Q1R(I + STQ1R)1STQ1

Lemma 1 Let U be such that QU = R. Then I + STU is invertible.

Proof: Suppose w Rk satisfies (I + STU)w = 0. Then,(Q + RST)U w = (QU)w + R(STU w) = Rw Rw = 0.

Q + RST being nonsingular, this gives U w = 0. Since rank (U) = k this implies w = 0. Theorem 1 If Qx0 = q and (I + STU)y = STx0 then x = x0

U y satisfies (Q + RST)x = q.

Proof: (Q + RST)(x0 U y) = Qx0 + RSTx0 QU y RSTU y= q + R(I + STU)y Ry RSTU y = q. 2

The solution x of (Q + RST)x = q is given by

x = x0 U y = Q1q Q1R(I + STU)1STx0= (Q1 Q1R(I + STQ1R)1STQ1)q.

Since Theorem 1 holds for all q Rn the Sherman-Morrison formula follows.Reference:W. W. Hager. Updating the inverse of a matrix. SIAM Rev., 31(2):221239, June 1989.



7/26

Ellipsoids

Let M be a (symmetric) positive definite n n matrix and z Rn. Then

EM,z :=

x : (x z)T

M(x z) 1denotes an ellipsoid centered at z. Note that

x EM,z

M1

2 (x z)

1.

Hence, putting u = M

1

2

(x z), we have x = z + M1

2

u and u 1. Therefore,EM,z =

x = z + M

1

2 u : uTu 1

.

In other words,

EM,z = z + M1

2 B(0, 1),

where B(0, 1) denotes the unit sphere, centered at the origin. The volume of B(0, 1) is given by

(n) =

n

2

n2

+ 1,

and the volume of EM,z by

vol

EM,z

=(n)

det(M).

Hence

ln volEM,z = ln (n) 1

2

ln det(M).



8/26

Ellipsoid method

z

a

Given is a set convex S. We want to find s

S. We assume

that an ellipsoid EM,z is given such that S EM,z.Step 1 : k = 0; Mk = M; zk = z;

Step 2 : if zk S: STOP;Step 3 : Find nonzero vector a such that

aTx aTzk, x S;

Step 4 : Construct smallest volume ellipsoid that contains

EM,z x Rn : aT(x zk) 0

;

Let this ellipsoid have matrix Mk+1 and centerzk+1.

Step 5 : k = k + 1;

Step 6 : Goto Step 2.



9/26

Basic construction (1)

Find m = (z, 0, . . . , 0), M = diag (a1, a2, . . . , an), such that the ellipsoid

EM,m = x Rn : (x m)TM(x m) 1contains B(0, 1) {x : x1 0} and has minimal volume. Note that

(x m)TM(x m) = a1(x1 z)2 +n

i=2

aix2i .

The unit vectors e1,

e2, . . . ,

en are on the boundary of B(0, 1)

{x : x1

0

}. We require them to lie

on the boundary of the ellipsoid. This gives:

a1(1 z)2 = 1a1z2 + ai = 1, 2 i n.

From this we obtain

a1 =1

(1 z)2 , ai = 1 a1z2 = 1 z

2

(1 z)2 =1 2z

(1 z)2 , i 2.

Recall that vol

EM,m

is minimal if det M is maximal. Since

det M = (1 2z)n

1

(1 z)2n ,

one may easily verify that this occurs if z = 1n+1

. Substituting this value we obtain

a1 = 1 + 1n

2

, ai = 1

1

n2, i

2.



10/26

Illustration for n = 2

-1 0 1

-1

0

1



11/26


a1 =(n + 1)2

n2, ai =

n2 1n2

, i 2.

B =

y : yTy 1 , H = {y : y1 0}M = n

21n2

I + 2

n1e1eT1

, z = 1

n+1e1

E = EM ,z

Theorem 2 (B H) E.Proof: One has y E if and only if

y e1

n + 1

Tn2 1

n2

I + 2e1eT1

n 1

y e1n + 1

1.

This is equivalent to

n2 1n

2

n

i=1

y2i +1

n2

+ y1 (y1

1)

2n + 2

n2

1.

Now let y B H. Then 0 y1 1 implies y1 (y1 1) 0. Alson

i=1 y2i 1. Since

n2 1n2

+1

n2= 1,

we obtain y E.



12/26


B = y : yTy 1M = n

21n2

I + 2n1e1e

T1

E = EM,z

Theorem 3 vol (E) < vol (B) e1

2(n+1) .

Proof: One has vol(E)vol(B)

=

det I

det M= 1

det M. Moreover,

det M =

n21

n2

n 1 + 2n1

=

n21

n2

n1 n+1

n

2.

Hence, using 1 + x

ex we get

1det M

=

n2

n21n1

nn+1

2=

1 + 1

n21n1

1 1n+12

en1

n21 e2

n+1 = e1

n+1 .

This implies the theorem. Optimization Group 12/26


13/26

Prototypical iteration

EM,z := x : (x z)T M(x

z)

1 = z + M

12B(0, 1)We know M, z, and a nonzero vector a. Define

M = n21n2

M + 2n1

aaT

aTM1a

, z = z + 1n+1

M1aaTM1a

By the Sherman-Morrison formula, one has

M1 = n2

n2 1

M1 2

n + 1

M1aaTM1aTM1a

Theorem 4

EM,z

x : aTx aTz

EM,z.

Theorem 5

vol EM,z < vol EM,z e1

2(n+1) .


P f f Th 5


14/26

Proof of Theorem 5

u := M1

2 a, b := u e1, R := 2(u + b)(u + b)T

u + b2 I

One has RT = R, Ru = b, R2 = I.Hence

M = n21n2

M + 2

n1aaT

aTM1a

= n

21n2

M1

2 I + 2n1M

12 aaTM

12

aTM1a M12= n

21n2

M1

2 R

I + 2n1

RM12 aaTM

12 R

aTM1a

RM

1

2

= n21n2

M1

2 R

I + 2n1

RM12 aaTM

12 R

aTM12 RRM

12 a

RM

1

2

= n21n2

M1

2 RI +2

n1e1eT1RM

1

2 .

Therefore,

det M =

n21n2

ndet M(det R)2 det

I + 2

n1e1eT1

=

n21n2

n 1 + 2

n1

det M,

and

det M

det M =n2 1n2

n 1 +

2

n 1

> e

1

n+1 .

Hence

vol

EM ,z

vol EM,z=

det Mdet M

< e1

2(n+1)

This proves Theorem 5.


P f f Th 4


15/26

Proof of Theorem 4

EM,z

x : aTx

aTz

EM,z.

Proof: Suppose (x z)TM(x z) 1 and aTx aTz. Putu = M

1

2 a, Ru = b =

u

e1, y := RM

1

2 (x

z)

Then

yTy = (x z)TM12 R2M12 (x z) = (x z)TM(x z) 1.Moreover,

u

y1 =

u

eT1 y = bTy = uTRu

= aTM1

2 RRM1

2 (x z) = aT(x z) 0.Now

(x z)TM(x z) =

(x z 1

n+1M1a

aTM1a)T

M(x z 1

n+1M1a

aTM1a) =(x z 1

n+1M1aaTM1a

)TM1

2 R

n21n2

I + 2

n1e1eT1

RM

1

2

x z M1a

aTM1a

=

y 1n+1

ue1u

I + 2

n1e1eT1

y 1

n+1ue1u

1.

The inequality follows just as in the proof of Theorem 1, since yTy 1 and y1 0.


T th


16/26

Two theorems

Theorem 6 Suppose we want to find a point in the set S, with S EM0,z0 and vol (S) > 0. Then thealgorithm will find a point in S after at most

2(n + 1) ln

vol

EM0,z0

vol (S)

iterations.

Proof: After k iterations, we havevol

EMk,zk

vol EM0,z0 e k2(n+1) .Since S EMk,zk for each k we obtain

vol (S)

vol EM0,z0 e k2(n+1) .

Hence

ln vol (S) ln vol

EM0,z0 k

2(n + 1),

which implies

k 2(n + 1) ln

vol

EM0,z0

vol (S)

.

This proves the theorem.


Two theorems (cont )


17/26

Two theorems (cont.)

Let

B(c, ) :=

{x :

x

c

}.

Theorem 7 Suppose we knowR such thatS B(0, R) and thatS contains a ballB (x, r) for somex andr > 0. Then the algorithm will find a point in S after at most

2n(n + 1 ) l n

Rr

iterations.

Proof: After k iterations, we have

vol (B (x, r)) vol (S) vol (B(0, R)) ek

2(n+1) .Taking logarithms again we get

ln vol (B (x, r)) ln vol (B(0, R)) k2(n + 1)

,

and hence

k 2(n + 1) ln

vol(B(0,R))vol(B(x,r))

= 2(n + 1) ln

(n)Rn

(n)rn

= 2(n + 1)n ln

Rr

.

This proves the theorem.


Two more theorems


18/26

Two more theorems

Actually, we can improve the previous two results a lot in terms of what we need to know.

Theorem 8 Suppose vol

S EM0,z0

> 0. Then the algorithm will find a point in S

after at most

2(n + 1) lnvol

EM0,z0

vol

S EM0,z0iterations.

Proof: Obvious.

Theorem 9 Suppose we know R and that S B(0, R) contains a ball B (x, r) for somex and r > 0. Then the algorithm will find a point in S after at most

2n(n + 1) ln

Rr

iterations.

Proof: Obvious. Optimization Group 18/26



19/26


Consider the minimization problem

z = mincTx : x S .

Let

P :=

x : x S, cTx z + .Theorem 10 Suppose we know EM0,z0 such that EM0,z0

P

=

. Then the algorithm will find a point in

P after at most 2(n + 1) ln

vol

EM0,z0

vol

EM0,z0

P

iterations.

Proof: Obvious. Theorem 11 Let us know R and B(0, R) P contains a ball B (x, r) for some x and r > 0. Then thealgorithm will find a point in P after at most

2(n + 1)n ln

Rr

iterations.

Proof: Obvious.


Optimization (cont )


20/26

Optimization (cont.)

Unfortunately we must know R, and this is most unfortunate. Let us fix this.

min

cTx : x S (H) mincTy

: y

S, > 0 .Note that the problem (H) at the right is homogeneous: if (y, ) is feasible (or optimal)

then so is (y,) for any > 0. Taking = 1/ we get a feasible (or optimal) solution

of (P).

The objective function in (H) is not convex. However, the level sets are convex and this is

enough for the ellipsoid method.

As far as y concerns, the feasible region is the cone generated by the convex set S, without

the origin (since > 0). Hence the (convex!) feasible region H is not closed, but also thisdoes not hurt the ellipsoid method.

Finally, by adding the constraint

(y, ) 1

we lose nothing, and the whole feasible region lies in B((0, 0), 1)!


Convexity of the homogeneous problem


21/26

Convexity of the homogeneous problem

Proposition 1 The set

(y, ) :

y

S, > 0is convex.

Proof: Let (y1, 1), (y2, 2) belong to the set. For 0

1, let (y, ) = (y1, 1) +

(1 )(y2, 2).y

=

y1 + (1 )y21 + (1 )2

=1

y11

+ (1 )2 y221 + (1 )2

= y11

+ (1 )y22

S,

where = 11+(1)2 .

Proposition 2 The level sets of cTy are convex.

Proof: For any z R one hascTy

z cTy z 0.

Here we used > 0. This implies the result. Optimization Group 21/26

The Key Proposition


22/26

The Key Proposition

We have seen that we can apply the ellipsoid method to the homogeneous problem (H), with

R = 1.

Bn denotes the unit sphere in Rn.

Proposition 3 Suppose the level set P contains a ball B(x, r). Let

HP :=

(y, ) :

y

P, > 0

.

Then

vol

Bn+1

vol

HP Bn+1 (n + 1)(n + 1) 1 + (r + x)

2n+12(n)rn

.

Theorem 12 Solving (P) via (H), starting at the unit ball Bn+1, the algorithm will findan -solution of (H) (and hence of (P)) after at most(n + 1)(n + 2) ln

1

r2+

1 +

xr

2+ 2(n + 2) ln

r

n

iterations.


Graphical illustration


23/26

G p c st t o

x

Bn+1


Proof of Proposition 3


24/26

p

HB(x, r) :=

(y, ) :y

B(x, r), > 0

.

The longest vector in the set

T :=

(y, ) :y

B(x, r), 1

HB(x, r)

is the vector

x1

+ r xx0 =

1 + r

x

x1 and has length

:= 1 + 1 +r

x2

x

2 = 1 + (r +

x

)

2.

Hence

1T Bn+1 HB(x, r) Bn+1 HP.Therefore,

vol (Bn+1 HP) vol

1T

= 1n+1

vol (T) = 1n+1

10

(n)nrnd

=(n)rn

(n + 1)n+1=

(n)rn

(n + 1) 1 + (r + x)2

n+1

2

.

Hence Proposition 3 follows. Optimization Group 24/26


25/26

Proof of Theorem 12


26/26

By Theorem 10 the algorithm needs at most

2(n + 2) ln

vol (Bn+1)

vol (HP Bn+1)iterations. Due to Proposition 3 this is

2(n + 2) ln

(n + 1)(n + 1)

1 + (r + x)2

n+1

2

(n)rn .

Since

(n + 1)(n + 1)

(n)=

(n + 1)n+1

2

n2

+ 1

n

2 n+1

2+ 1

n,

we obtain (omitting the brackets) the bound

2(n + 2) ln

n(1+(r+x)2)n+1

2

rn

= 2(n + 2) ln rn 1r

2 + 1 + xr2

n+1

2

= (n + 1)(n + 2) ln

1r2

+

1 +x

r

2+ 2(n + 2) ln

r

n

.

This completes the proof.


ellipsoid 4

Documents