b calibrating rank constrained correlation a majorized ... · calibrating rank constrained...
TRANSCRIPT
![Page 1: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/1.jpg)
b
b
b
b
b
bb
b
bb b
b
b
bb
b
b
b
b
b
b
b
b
b
b
b
b
A Majorized Penalty Approach forCalibrating Rank Constrained Correlation
Matrix Problems
Defeng Sun
Department of Mathematics and Risk Management Institute
National University of Singapore
This is a joint work with Yan Gao at NUS
July 19, 2010
![Page 2: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/2.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 2 / 48
To use a low rank matrix to approximate a given matrix dates back to E.Schmidt [Math. Ann. 63 (1907), pp. 433–476] and C. Eckart and G.Young [Psychometrika 1 (1936), pp. 211–218]:
min1
2‖X − Z‖2
F
s.t. rank(X) ≤ r
admits an analytic solution for a given Z ∈ ℜm×n (m ≤ n without lossof generality):
X∗ =r
∑
i=1
σi(Z)uivTi ,
where Z has the following singular value decomposition (SVD):
Z = U [diag(σ(Z)) 0]V T , σ1(Z) ≥ σ2(Z) ≥ . . . ≥ σm(Z) ≥ 0 .
![Page 3: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/3.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 3 / 48
On October 27, 2009, I received this from Universiteit van Tilburg:
My thesis is about correlations in a pension fund pooling. It is importantfor economic capital calculations. For some risks such as operationalrisk, I dont have data and hence I need to consult for an expert opinion.Then I might end up with not PSD matrices. Therefore, I need tocalculate nearest correlation matrix.
In my given correlation matrix, I want to fix the correlations, which aredata driven and I want the rest of the correlations not smaller than 0.1from original matrix.
Your code is very convenient for my study. However, ...
![Page 4: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/4.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 4 / 48
On November 3, 2009:
Thank you for your valuable time, comments and helping me aboutsolving my problem.
I gave no chance that my fixed constraints could be non-PSD before.Your advice solves the problem. I will modify my study in the light of it.
![Page 5: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/5.jpg)
The model
ICOCO2010, Guiyang, Guizhou NUS/SUN – 5 / 48
In this talk, we are interested in the following rank constrainedcovariance matrix problem
min ‖H ◦ (X − G)‖2F
s.t. Xii = 1, i = 1, . . . , n
Xij = eij, (i, j) ∈ Be ,
Xij ≥ lij, (i, j) ∈ Bl ,
Xij ≤ uij, (i, j) ∈ Bu ,
X ∈ Sn+ ,
rank(X) ≤ r ,
(1)
where Be, Bl, and Bu are three index subsets of {(i, j) | 1 ≤ i < j ≤ n}satisfying Be ∩Bl = ∅, Be ∩Bu = ∅, and lij < uij for any (i, j) ∈ Bl ∩Bu.
![Page 6: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/6.jpg)
continued
ICOCO2010, Guiyang, Guizhou NUS/SUN – 6 / 48
Here Sn and Sn+ are, respectively, the space of n × n symmetric
matrices and the cone of positive semidefinite matrices in Sn.
‖ · ‖F is the Frobenius norm defined in Sn.
H ≥ 0 is a weight matrix.
• Hij is larger if Gij is better estimated.
• Hij = 0 if Gij is missing.
A matrix X ∈ Sn is called a correlation matrix if X � 0 (i.e., X ∈ Sn+)
and Xii = 1, i = 1, . . . , n.
![Page 7: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/7.jpg)
A simple correlation matrix model
ICOCO2010, Guiyang, Guizhou NUS/SUN – 7 / 48
min ‖H ◦ (X − G)‖2F
s.t. Xii = 1 , i = 1, . . . , n
X � 0 ,
rank(X) ≤ r .
(2)
![Page 8: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/8.jpg)
The simplest corr. matrix model
ICOCO2010, Guiyang, Guizhou NUS/SUN – 8 / 48
min ‖(X − G)‖2F
s.t. Xii = 1 , i = 1, . . . , n
X � 0 ,
rank(X) ≤ r .
(3)
![Page 9: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/9.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 9 / 48
In finance and statistics, correlation matrices are in many situationsfound to be inconsistent, i.e., X � 0.
These include, but are not limited to,
■ Structured statistical estimations; data come from different timefrequencies
■ Stress testing regulated by Basel II;
■ Expert opinions in reinsurance, and etc.
![Page 10: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/10.jpg)
One correlation matrix
ICOCO2010, Guiyang, Guizhou NUS/SUN – 10 / 48
Partial market data1
G =
1.0000 0.9872 0.9485 0.9216 −0.0485 −0.0424
0.9872 1.0000 0.9551 0.9272 −0.0754 −0.0612
0.9485 0.9551 1.0000 0.9583 −0.0688 −0.0536
0.9216 0.9272 0.9583 1.0000 −0.1354 −0.1229
−0.0485 −0.0754 −0.0688 −0.1354 1.0000 0.9869
−0.0424 −0.0612 −0.0536 −0.1229 0.9869 1.0000
The eigenvalues of G are: 0.0087, 0.0162, 0.0347, 0.1000, 1.9669, and3.8736.
1RiskMetrics (www.riskmetrics.com/stddownload edu.html)
![Page 11: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/11.jpg)
Stress tested
ICOCO2010, Guiyang, Guizhou NUS/SUN – 11 / 48
Let’s change G to
[change G(1, 6) = G(6, 1) from −0.0424 to −0.1000]
1.0000 0.9872 0.9485 0.9216 −0.0485 −0.1000
0.9872 1.0000 0.9551 0.9272 −0.0754 −0.0612
0.9485 0.9551 1.0000 0.9583 −0.0688 −0.0536
0.9216 0.9272 0.9583 1.0000 −0.1354 −0.1229
−0.0485 −0.0754 −0.0688 −0.1354 1.0000 0.9869
−0.1000 −0.0612 −0.0536 −0.1229 0.9869 1.0000
The eigenvalues of G are: −0.0216, 0.0305, 0.0441, 0.1078, 1.9609, and3.8783.
![Page 12: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/12.jpg)
Missing data
ICOCO2010, Guiyang, Guizhou NUS/SUN – 12 / 48
On the other hand, some correlations may not be reliable or even missing:
G =
1.0000 0.9872 0.9485 0.9216 −0.0485 −−−0.9872 1.0000 0.9551 0.9272 −0.0754 −0.0612
0.9485 0.9551 1.0000 0.9583 −0.0688 −0.0536
0.9216 0.9272 0.9583 1.0000 −0.1354 −0.1229
−0.0485 −0.0754 −0.0688 −0.1354 1.0000 0.9869
−−− −0.0612 −0.0536 −0.1229 0.9869 1.0000
![Page 13: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/13.jpg)
Drop the rank constraint
ICOCO2010, Guiyang, Guizhou NUS/SUN – 13 / 48
Let us first consider the problem without the rank constraint:
min1
2‖H ◦ (X − G)‖2
F
s.t. Xii = 1 , i = 1, . . . , n
X � 0 .
(4)
When H = E, the matrix of ones, we get
min1
2‖X − G‖2
F
s.t. Xii = 1 , i = 1, . . . , n
X � 0 .
(5)
which is known as the nearest correlation matrix (NCM) problem, aterminology coined by Nick Higham (2002).
![Page 14: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/14.jpg)
The story starts
ICOCO2010, Guiyang, Guizhou NUS/SUN – 14 / 48
The NCM problem is a special case of the best approximation problem
min1
2‖x − c‖2
s.t. Ax ∈ b + Q ,
x ∈ K ,
where X is a real Hilbert space equipped with a scalar product 〈·, ·〉 andits induced norm ‖ · ‖, A : X → ℜm is a bounded linear operator,Q = {0}p × ℜq
+ is a polyhedral convex cone, 1 ≤ p ≤ m, q = m − p,and K is a closed convex cone in X .
![Page 15: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/15.jpg)
The KKT conditions
ICOCO2010, Guiyang, Guizhou NUS/SUN – 15 / 48
The Karush-Kuhn-Tucker conditions are
(x − z) − c −A∗y = 0
Q∗ ∋ y ⊥ Ax − b ∈ Q
K∗ ∋ z ⊥ x ∈ K ,
,
where “⊥” means the orthogonality. Q∗ is the dual cone of Q and K∗ isthe dual cone of K.
![Page 16: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/16.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 16 / 48
Equivalently,
(x − z) − c −A∗y = 0
Q∗ ∋ y ⊥ Ax − b ∈ Q
x − ΠK(x − z) = 0
,
where ΠK(x) is the unique optimal solution to
min1
2‖u − x‖2
s.t. u ∈ K .
![Page 17: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/17.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 17 / 48
Consequently, by first eliminating (x − z) and then x, we get
Q∗ ∋ y ⊥ AΠK(c + A∗y) − b ∈ Q ,
which is equivalent to
F (y) := y − ΠQ∗ [y − (AΠK(c + A∗y) − b)] = 0, y ∈ ℜm .
![Page 18: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/18.jpg)
The dual formulation
ICOCO2010, Guiyang, Guizhou NUS/SUN – 18 / 48
The above is nothing but the first order optimality condition to theconvex dual problem
max −θ(y) := −
[
1
2‖ΠK(c + A∗y)‖2 − 〈b, y〉 −
1
2‖c‖2
]
s.t. y ∈ Q∗ .
Then F can be written as
F (y) = y − ΠQ∗(y −∇θ(y)) .
![Page 19: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/19.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 19 / 48
Now, we only need to solve
F (y) = 0, y ∈ ℜm .
However, the difficulties are:
■ F is not differentiable at y;
■ F involves two metric projection operators;
■ Even if F is differentiable at y, it is too costly to compute F ′(y).
![Page 20: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/20.jpg)
The NCM problem
ICOCO2010, Guiyang, Guizhou NUS/SUN – 20 / 48
For the nearest correlation matrix problem,
• A(X) = diag(X), a vector consisting of all diagonal entries of X..
• A∗(y) = diag(y), the diagonal matrix.
• b = e, the vector of all ones in ℜn and K = Sn+.
Consequently, F can be written as
F (y) = AΠSn
+(G + A∗y) − b.
![Page 21: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/21.jpg)
The projector
ICOCO2010, Guiyang, Guizhou NUS/SUN – 21 / 48
For n = 1, we have
x+ := ΠS1+(x) = max(0, x).
Note that• x+ is only piecewise linear, but not smooth.• (x+)2 is continuously differentiable with
∇{1
2(x+)2
}
= x+,
but is not twice continuously differentiable.
![Page 22: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/22.jpg)
The one dimensional case
ICOCO2010, Guiyang, Guizhou NUS/SUN – 22 / 48
![Page 23: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/23.jpg)
The multi-dimensional case
ICOCO2010, Guiyang, Guizhou NUS/SUN – 23 / 48
The projector for K = Sn+:
x
x
Convex Cone
x2 3
1
ΠK
η
(η)K
0
![Page 24: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/24.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 24 / 48
Let X ∈ Sn have the following spectral decomposition
X = PΛP T ,
where Λ is the diagonal matrix of eigenvalues of X and P is acorresponding orthogonal matrix of orthonormal eigenvectors.
Then
X+ := ΠSn
+(X) = PΛ+P T .
![Page 25: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/25.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 25 / 48
We have
• ‖X+‖2 is continuously differentiable with
∇(1
2‖X+‖
2)
= X+,
but is not twice continuously differentiable.
• X+ is not piecewise smooth, but strongly semismooth2.
2 D.F. Sun and J. Sun. Semismooth matrix valued functions. Mathematics of
Operations Research 27 (2002) 150–169.
![Page 26: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/26.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 26 / 48
A quadratically convergent Newton’s method is then designed by Qi andSun3. The written code is called CorNewton.m.
"This piece of research work is simply great and
practical. I enjoyed reading your paper." –March 20, 2007, a home loan financial institution based inMcLean, VA.
"It’s very impressive work and I’ve also run theMatlab code found in Defeng’s home page. Itworks very well."– August 31, 2007, a major investmentbank based in New York city.
3H.D. Qi and D.F. Sun, A quadratically convergent Newton method for comput-ing the nearest correlation matrix. SIAM Journal on Matrix Analysis and Applications
28 (2006), pp. 360–385.
![Page 27: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/27.jpg)
Inequality constraints
ICOCO2010, Guiyang, Guizhou NUS/SUN – 27 / 48
If we have lower and upper bounds on X, F takes the form
F (y) = y − ΠQ∗ [y − (AΠSn
+(G + A∗y) − b)] ,
which involves double layered projections over convex cones.
A quadratically convergent inexact smoothing Newton-BICGStab methodis designed by Gao and Sun4.
Again, highly efficient.
4Y. Gao and D.F. Sun, Calibrating least squares covariance matrix problems withequality and inequality constraints, SIAM Journal on Matrix Analysis and Applications31 (2009), pp. 1432–1457.
![Page 28: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/28.jpg)
Back to the rank constraint
ICOCO2010, Guiyang, Guizhou NUS/SUN – 28 / 48
min1
2‖H ◦ (X − G)‖2
F
s.t. AX ∈ b + Q ,
X ∈ Sn+ ,
rank(X) ≤ k,
equivalently,
min1
2‖H ◦ (X − G)‖2
F
s.t. AX ∈ b + Q ,
X ∈ Sn+ ,
λi(X) = 0, i = k + 1, . . . , n.
![Page 29: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/29.jpg)
The penalty approach
ICOCO2010, Guiyang, Guizhou NUS/SUN – 29 / 48
Given c > 0, we consider a penalized version
min1
2‖H ◦ (X − G)‖2
F + c
n∑
i=k+1
λi(X)
s.t. AX ∈ b + Q ,
X ∈ Sn+ ,
or equivalently
min fc(X) :=1
2‖H ◦ (X − G)‖2
F + c〈I,X〉 − c
k∑
i=1
λi(X)
s.t. AX ∈ b + Q ,
X ∈ Sn+ .
![Page 30: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/30.jpg)
Majorization functions
ICOCO2010, Guiyang, Guizhou NUS/SUN – 30 / 48
Let h(X) :=∑k
i=1 λi(X) − 〈I,X〉. Since h is a convex function, forgiven Xk, we have
h(X) ≥ hk(X) := h(Xk) + 〈V k, X − Xk〉,
where V k ∈ ∂h(Xk). Thus, −h is majorized by −hk.
Let d ∈ ℜn be a positive vector such that
H ◦ H ≤ ddT .
For example, d = max(Hij)e. Let D1/2 = diag(d0.51 , . . . , d0.5
n ).
![Page 31: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/31.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 31 / 48
Let
g(X) :=1
2‖H ◦ (X − G)‖2
F .
Then g is majorized by
gk(X) := g(Xk)+〈H ◦H(Xk−G), X−Xk〉+1
2‖D1/2(X−Xk)D1/2‖2
F .
Thus, at Xk, fc is majorized by
fc(X) ≤ fk(X) := gk(X) − chk(X)
and fc(Xk) = fk(Xk).
![Page 32: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/32.jpg)
The idea of majorization
ICOCO2010, Guiyang, Guizhou NUS/SUN – 32 / 48
Instead of solving the penalized problem, the idea of the majorization isto solve, for given Xk, the following problem
min fkc (X) = gk(X) − chk(X)
s.t. AX ∈ b + Q ,
X ∈ Sn+ ,
which is a diagonal weighted least squares correlation matrix problem
min1
2‖D1/2(X − Xk)D1/2‖2
F
s.t. AX ∈ b + Q ,
X ∈ Sn+ .
![Page 33: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/33.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 33 / 48
Now, we can use the two Newton methods introduced earlier for themajorized subproblems!
fc(Xk+1) < fc(X
k) < · · · < fc(X1).
![Page 34: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/34.jpg)
Where is the rank condition?
ICOCO2010, Guiyang, Guizhou NUS/SUN – 34 / 48
Looks good? But how can one guarantee that we can get a final X∗
such that its rank is less or equal to k?
The answer is: increase c. That is, to have a sequence of {ck} withck+1 ≥ ck.
Will it work? Numerical stability? Does not need a large ck in numericalcomputations.
There are no known methods that can solve the general rank constrainedproblem. For the H-normed correlation matrix problems (withoutconstraints on the off diagonal entries), the major.m of R. Pietersz andJ.F. Groenen (2004) is the most efficient one so far [write X = Y Y T forY ∈ ℜn×k and apply component-by-component majorization.].
![Page 35: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/35.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 35 / 48
Let Y ∈ Sn be arbitrarily chosen. Suppose that Y has the spectraldecomposition
Y = UΣ(Y )UT , (6)
where U ∈ On is a corresponding orthogonal matrix of orthonormaleigenvectors of Y and Σ(Y ) := diag(σ(Y )) whereσ(Y ) = (σ1(Y ), . . . , σn(Y ))T is the column vector containing all theeigenvalues of Y being arranged in the non-increasing order in terms oftheir absolute values, i.e.,
|σ1(Y )| ≥ · · · ≥ |σn(Y )| ,
and whenever the equality holds, the larger one comes first, i.e.,
if |σi(Y )| = |σj(Y )| and σi(Y ) > σj(Y ), then i < j .
![Page 36: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/36.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 36 / 48
Define
α := {i | |σi(Y )| > |σr(Y )| }, β := {i | |σi(Y )| = |σr(Y )| },
γ := {i | |σi(Y )| < |σr(Y )| },
and β+ := {i | σi(Y ) = |σr(Y )| }, β− := {i | σi(Y ) = −|σr(Y )| } .
Denote
Ψr(Y ) := min1
2‖Z − Y ‖2
s.t. Z ∈ Sn(r) .(7)
Denote the set of optimal solutions to (7) by ΠSn(r)(Y ).
![Page 37: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/37.jpg)
Projection onto Sn(r)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 37 / 48
Lemma 1. Let Y ∈ Sn have the spectral decomposition as in (6). Thenthe solution set ΠSn(r)(Y ) to problem (7) can be characterized as follows
ΠSn(r)(Y ) =
{
[Uα UβQβ Uγ ]diag(v) [Uα UβQβ Uγ ]T∣
∣
∣
v ∈ V , Qβ =
[
Qβ+ 00 Qβ−
]
, Qβ+ ∈ O|β+|, Qβ− ∈ O|β−|
}
,
(8)where
V :={
v ∈ ℜn | vi = σi(Y ) for i ∈ α ∪ β1, vi = 0 for i ∈ (β \ β1) ∪ γ ,
where β1 ⊆ β and |β1| = r − |α|}
.
(9)
![Page 38: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/38.jpg)
Global Optimality Checking
ICOCO2010, Guiyang, Guizhou NUS/SUN – 38 / 48
Theorem 1. 5 The optimal solution (y, Y ) ∈ Q∗ × Sn to the the dualproblem satisfies
b −AΠSn
+
(
C + A∗y + Y)
∈ NQ∗(y) (10)
andΠSn
+
(
C + A∗y + Y)
∈ conv{
ΠSn(r)(C − Y )}
, (11)
where ΠSn(r)(·) is defined as in Lemma 1. Furthermore, if there exists amatrix X ∈ ΠSn(r)(C − Y ) such that X = ΠSn
+
(
C + A∗y + Y)
, then X
and (y, Y ) globally solve the primal problem with H = E and thecorresponding dual problem, respectively and there is no duality gapbetween the primal and dual problems.
5Y. Gao and D.F. Sun, A majorized penalty approach for calibrating rank con-strained correlation matrix problems, manuscript, March 2010.
![Page 39: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/39.jpg)
Testing Examples
ICOCO2010, Guiyang, Guizhou NUS/SUN – 39 / 48
The testing examples to be reported are given below.
Example 1. Let n = 500 and the weight matrix H = E. Fori, j = 1, . . . , n, Cij = 0.5 + 0.5e−0.05|i−j|. The index sets areBe = Bl = Bu = ∅.
Example 2. Let n = 500 and the weight matrix H = E. The matrix C
is extracted from the correlation matrix which is based on a 10, 000 genemicro-array data set obtained from 256 drugs treated rat livers. Theindex sets are Be = Bl = Bu = ∅.
Example 3. Let n = 500. The matrix C is the same as in Example 1,i.e., C = 0.5 + 0.5e−0.05|i−j| for i, j = 1, . . . , n. The index sets areBe = Bl = Bu = ∅. The weight matrix H is generated in the way suchthat all its entries are uniformly distributed in [0.1, 10] except for 2× 100entries in [0.01, 100].
![Page 40: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/40.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 40 / 48
Example 4. Let n = 500. The matrix C is the same as in Example 2.The index sets are Be = Bl = Bu = ∅ . The weight matrix H isgenerated in the same way as in Example 3.
Example 5. The matrix C is obtained from the gene data sets withdimension n = 1, 000 as in Example 2. The weight matrix H is the sameas in Example 3. The index sets Be, Bl, andBu ⊂ {(i, j) | 1 ≤ i < j ≤ n} consist of the indices of min(nr, n − i)randomly generated elements at the ith row of X, i = 1, . . . , n withnr = 5 for Be and nr = 10 for Bl and Bu. We take eij = 0 for(i, j) ∈ Be, lij = −0.1 for (i, j) ∈ Bl and uij = 0.1 for (i, j) ∈ Bu.
![Page 41: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/41.jpg)
Numerical Results
ICOCO2010, Guiyang, Guizhou NUS/SUN – 41 / 48
20 40 60 80 100 120
101
102
103
rank
time
(se
cs)
Example 5.1: n=500, H=E
PenCorr Major SemiNewton Dual−BFGS
20 40 60 80 100 12010
−15
10−10
10−5
100
rank
rela
tive
ga
p
Eample 5.1: n=500, H=E
PenCorr Major SemiNewton Dual−BFGS
![Page 42: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/42.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 42 / 48
Eg1 Major SemiNewton Dual-BFGS PenCorr
rank time residue relgap time residue relgap time residue relgap time residue relgap
2 1.9 1.564e2 3.4e-3 63.0 1.564e2 3.5e-3 432.0 1.660e2 6.5e-2 25.7 1.564e2 3.4e-35 2.2 7.883e1 6.5e-5 23.5 7.883e1 2.8e-5 24.6 7.883e1 1.1e-15 7.5 7.883e1 7.0e-510 2.7 3.869e1 6.9e-5 19.0 3.868e1 8.0e-6 8.0 3.868e1 1.7e-14 4.4 3.869e1 6.7e-515 4.2 2.325e1 8.3e-5 18.5 2.324e1 7.3e-6 6.0 2.324e1 3.4e-14 3.9 2.325e1 7.9e-520 7.5 1.571e1 8.8e-5 15.3 1.571e1 7.6e-6 5.6 1.571e1 2.9e-14 4.1 1.571e1 6.9e-525 12.8 1.145e1 1.1e-4 14.4 1.145e1 8.6e-6 5.0 1.145e1 1.8e-13 3.2 1.145e1 1.0e-430 19.4 8.797e0 1.3e-4 14.0 8.796e0 9.5e-6 4.3 8.795e0 4.4e-13 3.0 8.796e0 9.4e-535 34.4 7.020e0 1.7e-4 14.0 7.019e0 1.0e-5 4.8 7.019e0 2.0e-13 4.7 7.019e0 2.8e-540 43.4 5.766e0 2.2e-4 1.3 5.774e0 1.7e-3 4.3 5.764e0 5.6e-13 3.0 5.765e0 3.9e-545 63.6 4.843e0 3.0e-4 1.3 4.849e0 1.6e-3 4.5 4.841e0 7.4e-13 3.0 4.841e0 4.2e-550 80.1 4.141e0 4.0e-4 1.4 4.146e0 1.6e-3 4.3 4.139e0 1.8e-12 1.8 4.139e0 6.8e-560 145.0 3.156e0 6.7e-4 1.4 3.158e0 1.4e-3 4.5 3.153e0 8.4e-13 1.6 3.154e0 8.4e-570 243.0 2.507e0 1.1e-3 1.4 2.507e0 1.3e-3 4.3 2.504e0 3.4e-12 1.6 2.504e0 1.0e-480 333.0 2.053e0 1.6e-3 1.5 2.052e0 1.2e-3 4.1 2.050e0 4.2e-12 1.6 2.050e0 1.2e-490 452.0 1.722e0 2.4e-3 1.6 1.720e0 1.2e-3 4.2 1.718e0 1.1e-11 1.7 1.718e0 1.4e-4100 620.0 1.471e0 3.3e-3 1.5 1.468e0 1.1e-3 4.3 1.467e0 3.3e-12 1.6 1.467e0 1.5e-4125 1180.0 1.055e0 6.8e-3 1.7 1.049e0 9.9e-4 4.2 1.048e0 1.0e-11 1.7 1.048e0 1.8e-4
Table 1: Numerical results for Example 1
![Page 43: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/43.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 43 / 48
20 40 60 80 100 12010
−1
100
101
102
103
Rank
Tim
e (
secs
)
Example 5.2: n=500, H=E
20 40 60 80 100 12010
−15
10−10
10−5
100
Rank
Re
lativ
e g
ap
Example 5.2: n=500, H=E
PenCorr Major SemiNewton Dual−BFGS
PenCorr Major SemiNewton Dual−BFGS
![Page 44: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/44.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 44 / 48
Eg2 Major SemiNewton Dual-BFGS PenCorr
rank time residue relgap time residue relgap time residue relgap time residue relgap
2 0.6 2.858e2 6.5e-4 54.4 2.860e2 1.5e-3 304.5 2.862e2 2.1e-3 37.2 2.859e2 8.2e-45 6.0 1.350e2 2.0e-3 38.2 1.358e2 8.1e-3 78.8 1.367e2 1.5e-2 99.2 1.351e2 2.4e-310 9.3 6.716e1 4.4e-4 32.7 6.735e1 3.2e-3 58.3 6.802e1 1.3e-2 32.1 6.719e1 9.7e-415 8.8 4.097e1 3.4e-4 26.8 4.100e1 1.0e-3 44.6 4.096e1 1.0e-4 18.4 4.099e1 7.5e-420 13.0 2.842e1 7.3e-4 18.8 2.844e1 1.4e-3 40.4 2.842e1 8.9e-4 16.6 2.843e1 1.1e-325 34.9 2.149e1 1.2e-3 18.0 2.152e1 2.6e-3 26.6 2.149e1 1.2e-3 16.4 2.151e1 2.2e-330 33.7 1.693e1 4.3e-4 17.3 1.695e1 1.7e-3 23.0 1.694e1 7.8e-4 14.5 1.694e1 1.2e-335 71.8 1.379e1 1.3e-3 18.1 1.381e1 2.6e-3 19.7 1.378e1 7.1e-4 11.9 1.379e1 1.6e-340 50.0 1.151e1 1.5e-3 12.5 1.152e1 2.1e-3 34.7 1.145e1 3.2e-4 7.7 1.151e1 1.6e-345 43.3 9.733e0 9.6e-4 10.6 9.736e0 1.3e-3 23.1 9.733e0 9.2e-4 6.3 9.733e0 1.0e-350 44.5 8.318e0 4.1e-4 10.7 8.319e0 4.8e-4 19.7 8.315e0 5.1e-6 5.7 8.318e0 4.5e-460 66.5 6.214e0 8.1e-4 10.9 6.214e0 7.4e-4 6.1 6.209e0 1.4e-13 6.9 6.213e0 5.9e-470 91.2 4.733e0 1.1e-3 11.0 4.731e0 8.2e-4 23.1 4.728e0 1.9e-4 4.6 4.731e0 7.2e-480 93.0 3.663e0 8.7e-4 2.2 3.800e0 3.8e-2 5.2 3.660e0 4.0e-13 2.9 3.662e0 4.5e-490 125.0 2.865e0 1.2e-3 2.0 2.962e0 3.5e-2 5.0 2.862e0 5.1e-13 3.0 2.864e0 7.0e-4100 150.0 2.255e0 1.4e-3 1.7 2.323e0 3.2e-2 15.1 2.254e0 7.8e-4 2.9 2.254e0 8.3e-4125 288.6 1.269e0 2.4e-3 1.4 1.304e0 3.0e-2 17.1 1.266e0 1.6e-4 2.7 1.268e0 1.4e-3
Table 2: Numerical results for Example 2
![Page 45: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/45.jpg)
ICOCO2010, Guiyang, Guizhou NUS/SUN – 45 / 48
Example 3 Example 4
Majorw PenCorr Majorw PenCorr
rank time residue time residue time residue time residue
2 8.8 1.805e2 81.2 1.804e2 2.9 3.274e2 141.6 3.277e25 27.0 8.984e1 70.0 8.986e1 34.4 1.523e2 245.0 1.522e210 38.7 4.382e1 48.7 4.383e1 48.5 7.423e1 98.7 7.428e115 55.5 2.616e1 43.7 2.618e1 70.5 4.442e1 79.9 4.446e120 84.4 1.751e1 39.1 1.753e1 101.4 2.985e1 67.0 2.987e125 117.0 1.265e1 38.2 1.266e1 289.6 2.197e1 69.8 2.204e130 171.8 9.657e0 36.5 9.657e0 335.6 1.694e1 65.8 1.699e135 250.6 7.639e0 39.8 7.632e0 436.7 1.345e1 71.0 1.343e140 324.7 6.213e0 38.8 6.203e0 470.7 1.098e1 50.5 1.098e145 408.4 5.169e0 38.4 5.148e0 498.7 9.104e0 47.7 9.094e050 502.2 4.391e0 37.5 4.355e0 639.5 7.625e0 48.0 7.623e060 654.1 3.290e0 35.6 3.219e0 837.6 5.552e0 44.0 5.523e070 972.5 2.579e0 38.2 2.481e0 987.5 4.135e0 44.9 4.084e080 1274.9 2.090e0 42.6 1.959e0 1212.0 3.127e0 38.0 3.082e090 1526.9 1.740e0 44.0 1.588e0 1417.0 2.393e0 35.6 2.345e0100 1713.7 1.478e0 40.9 1.310e0 1612.0 1.865e0 32.7 1.814e0125 2438.1 1.052e0 44.6 8.591e-1 1873.0 1.030e0 27.7 9.748e-1
Table 3: Numerical results for Example 3 and 4
![Page 46: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/46.jpg)
A general example
ICOCO2010, Guiyang, Guizhou NUS/SUN – 46 / 48
Example 5 PenCorr
rank time residue20 11640.0 1.872e250 1570.0 1.011e2100 899.0 8.068e1250 318.3 7.574e1500 326.3 7.574e1
Table 4: Numerical results for Example 5
![Page 47: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/47.jpg)
Final remarks
ICOCO2010, Guiyang, Guizhou NUS/SUN – 47 / 48
• A code named PenCorr.m can efficiently solve all sorts of rankconstrained correlation matrix problems. Faster when rank is larger.
• The techniques may be used to solve other problems, e.g., lowrank matrix problems with sparsity.
• The limitation is that it cannot solve problems for matricesexceeding the dimension 4, 000 by 4, 000 on a PC due to memoryconstraints.
![Page 48: b Calibrating Rank Constrained Correlation A Majorized ... · Calibrating Rank Constrained Correlation Matrix Problems Defeng Sun Department of Mathematics and Risk Management Institute](https://reader036.vdocument.in/reader036/viewer/2022062509/60f8a26e63a22e144a78073a/html5/thumbnails/48.jpg)
End of talk
ICOCO2010, Guiyang, Guizhou NUS/SUN – 48 / 48
Thank you! :)