simultaneous diagonalizationeaton.math.rpi.edu/faculty/mitchell/courses/matp...2 ˜ 3 i 2 4 ⇤ 100...
TRANSCRIPT
Simultaneous Diagonalization
John E. Mitchell
Department of Mathematical SciencesRPI, Troy, NY 12180 USA
April 2018
Mitchell Simultaneous Diagonalization 1 / 22
SDP formulation
Outline
1 SDP formulation
2 Simultaneous diagonalization
3 Simultaneous diagonalization of optimal solutions
4 Strict complementarity
Mitchell Simultaneous Diagonalization 2 / 22
SDP formulation
Primal SDPWe write our standard form semidefinite program as
minX C • Xsubject to Ai • X = bi , i = 1, . . . ,m
X ⌫ 0(1)
where:
C is an n ⇥ n symmetric matrixAi is an n ⇥ n symmetric matrix for i = 1, . . . ,mbi is a scalar for i = 1, . . . ,m.
The parameter matrices C and Ai need not be positive semidefinite,although they are assumed to be symmetric. Recall that C • Xrepresents the Frobenius inner product between the symmetricmatrices C and X , which is equal to the trace(CX ).
Mitchell Simultaneous Diagonalization 3 / 22
SDP formulation
Dual SDP
The dual problem is
maxy2IRm,S2Sn+
bT ysubject to
Pmi=1 yiAi + S = C
S ⌫ 0(2)
Notice that the dual slack variables S = C �Pm
i=1 yiAi constitute asymmetric positive semidefinite matrix in feasible dual solutions.
Mitchell Simultaneous Diagonalization 4 / 22
Simultaneous diagonalization
Outline
1 SDP formulation
2 Simultaneous diagonalization
3 Simultaneous diagonalization of optimal solutions
4 Strict complementarity
Mitchell Simultaneous Diagonalization 5 / 22
Simultaneous diagonalization
Simultaneous diagonalization
Two symmetric n ⇥ n matrices are simultaneously diagonalizable ifthey have the same eigenvectors.
LemmaIf the n ⇥ n symmetric matrices M and R are simultaneouslydiagonalizable then they commute.
Mitchell Simultaneous Diagonalization 6 / 22
1412=1211.
Simultaneous diagonalization
Proof of lemma
Let the columns of the orthogonal matrix P consist of the eigenvectorsof the matrices, so
M = P⇤PT , R = P⌅PT
for two diagonal matrices ⇤ and ⌅. Then
MR = P⇤PT P⌅PT
= P⇤⌅PT since P is orthogonal= P⌅⇤PT since diagonal matrices commute= P⌅PT P⇤PT since P is orthogonal= RM
as required.
Mitchell Simultaneous Diagonalization 7 / 22
A : diagonal.din, entries
a r eeigenvalues.
- f t p . E - p t pA O F-fr...-in)
e .I- I l
I I l
'f";D"..-11"":*.)
Simultaneous diagonalization
The converseThe converse also holds, so symmetric matrices commute if and only ifthey are simultaneously diagonalizable.
Sketch of proof of converse:Assume symmetric R, M commute.Assume R = UDUT , so diagonal entries of D are eigenvalues of R,and columns of U are eigenvectors of R. Then
RM = MR =) UDUT M = MUDUT =) DUT MU = UT MUD
since U orthogonal. For (i , j)-entry, Dii(UMUT )ij = (UMUT )ijDjj , so
UMUT =
2
6664
⇤ 0 . . . 00 ⇤ 0... . . . ...0 0 . . . ⇤
3
7775
where each block correspondsto a different eigenvalue Dii .Diagonalize M by diagonalizingdiagonal blocks / eigenspaces of R
Mitchell Simultaneous Diagonalization 8 / 22
-
ti¥Duu=u¥@Eu)
- I I 1 ) G )
I -
I-
Simultaneous diagonalization of optimal solutions
Outline
1 SDP formulation
2 Simultaneous diagonalization
3 Simultaneous diagonalization of optimal solutions
4 Strict complementarity
Mitchell Simultaneous Diagonalization 9 / 22
Simultaneous diagonalization of optimal solutions
Duality gap
Let X be feasible in (1) and (y ,S) be feasible in (2). The duality gap is
C • X � bT y =
mX
i=1
yiAi + S
!• X � bT y
= S • X +mX
i=1
yi(Ai • X )� bT y
= S • X + bT y � bT y= S • X .
Mitchell Simultaneous Diagonalization 10 / 22
M i r Cox m a x blys t . Aiox--bi f t . S=C-Ey.-Ai
X i a o S E O
n e
⇒ i
=
Simultaneous diagonalization of optimal solutions
Frobenius product of X ⇤ and S⇤
Assume (1) and (2) have optimal solutions and their values agree.
Let X ⇤ solve (1) and (y⇤,S⇤) solve (2).
From the argument above, the Frobenius product
trace(S⇤X ⇤) = S⇤ • X ⇤ = 0. (3)
Mitchell Simultaneous Diagonalization 11 / 22
-Sun of alia, entries of
(5×44) i s z e r o .
Simultaneous diagonalization of optimal solutions
Eigenvectors of optimal solutions
We can actually say something far stronger,namely the matrix product S⇤X ⇤ = 0, the zero matrix.
So it is not just the trace of S⇤X ⇤ that is equal to zero.
TheoremLet X ⇤ solve (1) and (y⇤,S⇤) solve (2) and assume S⇤ • X ⇤ = 0. ThenS⇤X ⇤ = 0 = X ⇤S⇤.
Mitchell Simultaneous Diagonalization 12 / 22
six#f:::-oil#
-
Simultaneous diagonalization of optimal solutions
Proof, part 1Since S⇤ 2 Sn
+ it has an eigendecomposition,
S⇤ = Q⇤QT ,
where the columns {qi , i = 1, . . . , n} of Q are the eigenvectors of S⇤
and the diagonal entries of the diagonal matrix ⇤ are the eigenvaluesof S⇤. We now have
0 = S⇤ • X ⇤
= trace(Q⇤QT X ⇤)
= trace(⇤QT X ⇤Q) from properties of trace
=nX
i=1
⇤ii
⇣QT X ⇤Q
⌘
iisince ⇤ is diagonal
=nX
i=1
⇤ii qTi X ⇤qi
Mitchell Simultaneous Diagonalization 13 / 22
india:}
trace(ACE'X*Q)→=§(gAijCQTMQ7jiJ-
[email protected]'t.ie;
-=§{QkiX*uQti
Simultaneous diagonalization of optimal solutions
Proof, part 2
Since X ⇤ is positive semidefinite, we have qTi X ⇤qi � 0 8i .
Since S⇤ is positive semidefinite, we have ⇤ii � 0 8i .Thus, we must have
⇤ii qTi X ⇤qi = 0 8i .
Hence, if qi is an eigenvector of S⇤ with a positive eigenvalue, we musthave
0 = qTi X ⇤qi = qT
i LLT qi = kLT qik2
where L is the Cholesky factor of X ⇤.
Thus, LT qi = 0, so X ⇤qi = 0. So qi is in the nullspace of X ⇤,so it is an eigenvector of X ⇤ with eigenvalue 0.
Mitchell Simultaneous Diagonalization 14 / 22
O = §Hi iqExqi- :3¥¥ 0 : X EO .
O- h i i - O
-l X l#EI :
-
LTgi= O = ) L Eqi i o ⇒ x * g i t0 = 0qi
Simultaneous diagonalization of optimal solutions
Proof, part 3
We order the columns of Q as Q = [Q1,Q2],where the columns of Q1 are eigenvectors with positive eigenvalueand the columns of Q2 are eigenvectors with an eigenvalue of 0.
We can write the eigendecomposition as
S⇤ =⇥
Q1 Q2⇤ ⇤1 0
0 0
� QT
1QT
2
�
where the diagonal entries of ⇤1 are positive.
Note that X ⇤Q1 = 0, since each eigenvector qi of S⇤ with positiveeigenvalue satisfies X ⇤q1 = 0.
Mitchell Simultaneous Diagonalization 15 / 22
-
= e aQ t
-i
Q,:[q,... gr) ,w i l t r e r a n4 ( s t )
XQ,=[Xq, Xq...Xqr)
Simultaneous diagonalization of optimal solutions
Proof, part 4
Thus,
X ⇤S⇤ = X ⇤ ⇥ Q1 Q2⇤ ⇤1 0
0 0
� QT
1QT
2
�
=⇥
0 X ⇤Q2⇤ ⇤1 0
0 0
� QT
1QT
2
�
=⇥
0 0⇤ QT
1QT
2
�
= 0, the n ⇥ n matrix of zeroes.
We also haveS⇤X ⇤ = (X ⇤S⇤)T = 0T = 0,
as required.
Mitchell Simultaneous Diagonalization 16 / 22
Im
- cyan①
since 54,#symmetric
Simultaneous diagonalization of optimal solutions
Simultaneous diagonalization
Note that S⇤ and X ⇤ commute, with both products X ⇤S⇤ = S⇤X ⇤ = 0.
Hence, the matrices are simultaneously diagonalizable.
In general, we can construct an orthonormal basis for IRn consisting ofthree sets of vectors:
Eigenvectors of S⇤ with positive eigenvalue that are in thenullspace of X ⇤. These eigenvectors comprise the columns of amatrix Q1.A basis for the intersection of the nullspaces of X ⇤ and S⇤, whichwe denote as the columns of a matrix Q̃2, andEigenvectors of X ⇤ with positive eigenvalue that are in thenullspace of S⇤. These eigenvectors comprise the columns of amatrix Q̃3.
Mitchell Simultaneous Diagonalization 17 / 22
-
Simultaneous diagonalization of optimal solutions
Simultaneous diagonalization
Note that S⇤ and X ⇤ commute, with both products X ⇤S⇤ = S⇤X ⇤ = 0.
Hence, the matrices are simultaneously diagonalizable.
In general, we can construct an orthonormal basis for IRn consisting ofthree sets of vectors:
Eigenvectors of S⇤ with positive eigenvalue that are in thenullspace of X ⇤. These eigenvectors comprise the columns of amatrix Q1.A basis for the intersection of the nullspaces of X ⇤ and S⇤, whichwe denote as the columns of a matrix Q̃2, andEigenvectors of X ⇤ with positive eigenvalue that are in thenullspace of S⇤. These eigenvectors comprise the columns of amatrix Q̃3.
Mitchell Simultaneous Diagonalization 17 / 22
Simultaneous diagonalization of optimal solutions
Simultaneous diagonalization
Note that S⇤ and X ⇤ commute, with both products X ⇤S⇤ = S⇤X ⇤ = 0.
Hence, the matrices are simultaneously diagonalizable.
In general, we can construct an orthonormal basis for IRn consisting ofthree sets of vectors:
Eigenvectors of S⇤ with positive eigenvalue that are in thenullspace of X ⇤. These eigenvectors comprise the columns of amatrix Q1.A basis for the intersection of the nullspaces of X ⇤ and S⇤, whichwe denote as the columns of a matrix Q̃2, andEigenvectors of X ⇤ with positive eigenvalue that are in thenullspace of S⇤. These eigenvectors comprise the columns of amatrix Q̃3.
Mitchell Simultaneous Diagonalization 17 / 22
Simultaneous diagonalization of optimal solutions
Writing S⇤ and X ⇤ using eigenvectors
We can then write
S⇤ =h
Q1 Q̃2 Q̃3
i2
4⇤1 0 00 0 00 0 0
3
5
2
4QT
1Q̃T
2Q̃T
3
3
5
and
X ⇤ =h
Q1 Q̃2 Q̃3
i2
40 0 00 0 00 0 ⇤3
3
5
2
4QT
1Q̃T
2Q̃T
3
3
5
where the diagonal entries in ⇤1 and ⇤3 are positive.
This is the SDP version of complementary slackness.
Mitchell Simultaneous Diagonalization 18 / 22
-
Strict complementarity
Outline
1 SDP formulation
2 Simultaneous diagonalization
3 Simultaneous diagonalization of optimal solutions
4 Strict complementarity
Mitchell Simultaneous Diagonalization 19 / 22
Strict complementarity
Strict complementarity for LP
In the linear programming case, there exist optimal solutions that arestrictly complementary.
The analogue of this in the SDP case is for
rank(X ⇤) + rank(S⇤) = n,
so there are no columns in Q̃2.
However, this does not always hold, as in the following example.
Mitchell Simultaneous Diagonalization 20 / 22
Strict complementarity
Example
The primal SDP is
minX2S3+
X33
subject to X11 = 1X13 + X22 + X31 = 0X12 + X33 + X21 = 0
X ⌫ 0
The unique optimal primal solution is
X ⇤ =
2
41 0 00 0 00 0 0
3
5 .
Mitchell Simultaneous Diagonalization 21 / 22
I f X , ,=0 : ⇒ XueXu--0,
[?§¥ 0= ) X , ]= K ,= Xs,= X} ,- 0
. = DX ,e-O.←
O
Strict complementarity
Dual of example
The dual problem is
maxy ,S y1
subject to S =
2
4�y1 �y3 �y2�y3 �y2 0�y2 0 1 � y3
3
5
S ⌫ 0.
The unique dual optimal solution is y⇤ = (0, 0, 0), giving
S⇤ =
2
40 0 00 0 00 0 1
3
5 .
Hence rank(X ⇤) = rank(S⇤) = 1 and rank(X ⇤)+rank(S⇤) = 2 < 3 = n.
Mitchell Simultaneous Diagonalization 22 / 22
f . c-Ey..Ai-18.%:/-g,foo!:b- y .fo,:{I-of