tensor decompositions, matrix completion and singular...
TRANSCRIPT
Tensor Decompositions, Matrix Completion andSingular Values
Harm Derksen
University of Michigan
ICERM, Computational Nonlinear AlgebraJune 3, 2014
Harm Derksen Tensors, LRMC and Singular Values
Tensor Product Spaces
F a fieldV (i) ∼= Fni F-vector space for i = 1, 2, . . . , dV = V (1) ⊗ V (2) ⊗ · · · ⊗ V (d) ∼= Fn1×···×nd tensor product space
Definition
A pure tensor is a tensor of the form v (1) ⊗ v (2) ⊗ · · · ⊗ v (d)
(v (i) ∈ V (i)).
Problem
Write a given tensor as a sum of the smallest number of puretensors.
(for F = R,C: Canonical Polyadic (CP) decompositions,PARAFAC, CANDECOMP)
Harm Derksen Tensors, LRMC and Singular Values
Tensor Product Spaces
F a fieldV (i) ∼= Fni F-vector space for i = 1, 2, . . . , dV = V (1) ⊗ V (2) ⊗ · · · ⊗ V (d) ∼= Fn1×···×nd tensor product space
Definition
A pure tensor is a tensor of the form v (1) ⊗ v (2) ⊗ · · · ⊗ v (d)
(v (i) ∈ V (i)).
Problem
Write a given tensor as a sum of the smallest number of puretensors.
(for F = R,C: Canonical Polyadic (CP) decompositions,PARAFAC, CANDECOMP)
Harm Derksen Tensors, LRMC and Singular Values
Applications
I psychometrics
I chemometrics
I algebraic complexity theory
I signal processing
I numerical linear algebra
I computer vision
I numerical analysis
I data mining
I graph analysis
I neuroscience
I economics/finance
Harm Derksen Tensors, LRMC and Singular Values
Application: Fluorescence Spectroscopy
We have p samples of mixtures of unknown chemical compounds.Every mixture is excited with light of m different wavelengths.Light of n wavelengths is emitted from the mixture . Measuringthe intensities, one obtains an m × n excitation-emisson matrix forevery sample. This yields a p ×m × n matrix, which is a 3-waytensor T .Every chemical compound corresponds to a rank 1 tensor. Bywriting T as the sum of r = rank(T ) pure tensors, we candistinguish r chemical compounds and find the excitation-emissionmatrix for each of them.
Harm Derksen Tensors, LRMC and Singular Values
Application:Fluorescence Spectroscopy(Image: Lei Li, Andrew Barron)
Harm Derksen Tensors, LRMC and Singular Values
Tensor rank
Definition
The rank of a tensor T ∈ V is the smallest nonnegative integer rsuch that we can write T as a sum of r pure tensors.
For example,
T = e1 ⊗ e1 ⊗ e1 + e1 ⊗ e2 ⊗ e2 + e2 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e1
has rank 2 because
T = 12(e1+e2)⊗(e1+e2)⊗(e1+e2)+ 1
2(e1−e2)⊗(e1−e2)⊗(e1−e2)
If d = 2 and V = V (1) ⊗ V (2) ∼= Rd1×d2 then the tensor rank isjust the rank of a matrix.
Harm Derksen Tensors, LRMC and Singular Values
Tensor rank
Definition
The rank of a tensor T ∈ V is the smallest nonnegative integer rsuch that we can write T as a sum of r pure tensors.
For example,
T = e1 ⊗ e1 ⊗ e1 + e1 ⊗ e2 ⊗ e2 + e2 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e1
has rank 2 because
T = 12(e1+e2)⊗(e1+e2)⊗(e1+e2)+ 1
2(e1−e2)⊗(e1−e2)⊗(e1−e2)
If d = 2 and V = V (1) ⊗ V (2) ∼= Rd1×d2 then the tensor rank isjust the rank of a matrix.
Harm Derksen Tensors, LRMC and Singular Values
Tensor rank
Definition
The rank of a tensor T ∈ V is the smallest nonnegative integer rsuch that we can write T as a sum of r pure tensors.
For example,
T = e1 ⊗ e1 ⊗ e1 + e1 ⊗ e2 ⊗ e2 + e2 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e1
has rank 2 because
T = 12(e1+e2)⊗(e1+e2)⊗(e1+e2)+ 1
2(e1−e2)⊗(e1−e2)⊗(e1−e2)
If d = 2 and V = V (1) ⊗ V (2) ∼= Rd1×d2 then the tensor rank isjust the rank of a matrix.
Harm Derksen Tensors, LRMC and Singular Values
Tensor rank
Definition
The rank of a tensor T ∈ V is the smallest nonnegative integer rsuch that we can write T as a sum of r pure tensors.
For example,
T = e1 ⊗ e1 ⊗ e1 + e1 ⊗ e2 ⊗ e2 + e2 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e1
has rank 2 because
T = 12(e1+e2)⊗(e1+e2)⊗(e1+e2)+ 1
2(e1−e2)⊗(e1−e2)⊗(e1−e2)
If d = 2 and V = V (1) ⊗ V (2) ∼= Rd1×d2 then the tensor rank isjust the rank of a matrix.
Harm Derksen Tensors, LRMC and Singular Values
Application: Algebraic Complexity Theory
V = Matn,n(F)⊗Matn,n(F)⊗Matn,n(F)
Tn =∑n
i ,j ,k=1 ei ,j ⊗ ej ,k ⊗ ek,i
rank(Tn) is the number of multiplications needed to multiply twon × n matrices. Clearly rank(Tn) ≤ n3.
Theorem
If rank(Tm) ≤ k , then rank(Tn) = O(nlogm(k)) and two n × nmatrices can be multiplied using O(nlogm(k)) arithmetic operations.
Harm Derksen Tensors, LRMC and Singular Values
Application: Algebraic Complexity Theory
V = Matn,n(F)⊗Matn,n(F)⊗Matn,n(F)
Tn =∑n
i ,j ,k=1 ei ,j ⊗ ej ,k ⊗ ek,i
rank(Tn) is the number of multiplications needed to multiply twon × n matrices. Clearly rank(Tn) ≤ n3.
Theorem
If rank(Tm) ≤ k , then rank(Tn) = O(nlogm(k)) and two n × nmatrices can be multiplied using O(nlogm(k)) arithmetic operations.
Harm Derksen Tensors, LRMC and Singular Values
Application: Algebraic Complexity Theory
Theorem (Strassen 1969)
rank(T2) ≤ 7, so rank(Tn) = O(nlog2(7)) = O(n2.8073...).
Theorem (Williams 2012)
rank(Tn) = O(n2.3727)
Theorem (Masseranti, Raviolo 2013)
rank(Tn) ≥ 3n2 − 2√
2n3/2 − 3n.
Harm Derksen Tensors, LRMC and Singular Values
Low Rank Matrix Completion
Problem
Given a partially filled matrix, complete the matrix such that theresulting matrix has the smallest possible rank.
For example −2 · ·2 3 ·· 6 2
can be completed to a rank 1 matrix−2 −3 −1
2 3 14 6 2
Harm Derksen Tensors, LRMC and Singular Values
Low Rank Matrix Completion
Problem
Given a partially filled matrix, complete the matrix such that theresulting matrix has the smallest possible rank.
For example −2 · ·2 3 ·· 6 2
can be completed to a rank 1 matrix−2 −3 −12 3 14 6 2
Harm Derksen Tensors, LRMC and Singular Values
Low Rank Matrix Completion
Problem
Given a partially filled matrix, complete the matrix such that theresulting matrix has the smallest possible rank.
For example −2 · ·2 3 ·· 6 2
can be completed to a rank 1 matrix−2 −3 −1
2 3 14 6 2
Harm Derksen Tensors, LRMC and Singular Values
Application: The Netflix Problem
The DVD rental company Netflix has 480,189 users, and 17,770movies. The user ratings for every user can be put in a480, 189× 17, 770 matrix A = (ai ,j) for which only few entries areknown (since most users have only seen a fraction of the 17,770movies).
Presumably, the rank of the matrix A is low. Using Low rankmatrix completion, Netflix can predict whether users like a moviethey have seen, and will recommend movies to the users.
Harm Derksen Tensors, LRMC and Singular Values
Reduction LRMC to Tensor Rank
A is n ×m matrixare allowed to change entries in positions (i1, j1), . . . , (is , js).mrank(A) minimal possible rank
Define
T =s∑
k=1
eik ⊗ ejk ⊗ ek + A⊗ es+1 ∈ Cn ⊗ Cm ⊗ Cs+1
Theorem (D.)
rank(T ) = mrank(A) + s
Harm Derksen Tensors, LRMC and Singular Values
Reduction LRMC to Tensor Rank
A is n ×m matrixare allowed to change entries in positions (i1, j1), . . . , (is , js).mrank(A) minimal possible rank
Define
T =s∑
k=1
eik ⊗ ejk ⊗ ek + A⊗ es+1 ∈ Cn ⊗ Cm ⊗ Cs+1
Theorem (D.)
rank(T ) = mrank(A) + s
Harm Derksen Tensors, LRMC and Singular Values
Example
A =
(1 t· 1
)For T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ (e1 + te2)⊗ e2 + e2 ⊗ e2 ⊗ e2 we haverank(T ) = 1 + mrank(A).
mrank(A) = 1 if t 6= 0 and mrank(A) = 2 if t = 0.
If t = 0, then T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e2 hasrank 3.
If t 6= 0 then rank(T ) = 2 and
T = e2 ⊗ e1 ⊗ (e1 − t−1e2) + (e1 + t−1e2)⊗ (e1 + te2)⊗ e2
Harm Derksen Tensors, LRMC and Singular Values
Example
A =
(1 t· 1
)For T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ (e1 + te2)⊗ e2 + e2 ⊗ e2 ⊗ e2 we haverank(T ) = 1 + mrank(A).
mrank(A) = 1 if t 6= 0 and mrank(A) = 2 if t = 0.
If t = 0, then T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e2 hasrank 3.
If t 6= 0 then rank(T ) = 2 and
T = e2 ⊗ e1 ⊗ (e1 − t−1e2) + (e1 + t−1e2)⊗ (e1 + te2)⊗ e2
Harm Derksen Tensors, LRMC and Singular Values
Example
A =
(1 t· 1
)For T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ (e1 + te2)⊗ e2 + e2 ⊗ e2 ⊗ e2 we haverank(T ) = 1 + mrank(A).
mrank(A) = 1 if t 6= 0 and mrank(A) = 2 if t = 0.
If t = 0, then T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e2 hasrank 3.
If t 6= 0 then rank(T ) = 2 and
T = e2 ⊗ e1 ⊗ (e1 − t−1e2) + (e1 + t−1e2)⊗ (e1 + te2)⊗ e2
Harm Derksen Tensors, LRMC and Singular Values
Example
A =
(1 t· 1
)For T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ (e1 + te2)⊗ e2 + e2 ⊗ e2 ⊗ e2 we haverank(T ) = 1 + mrank(A).
mrank(A) = 1 if t 6= 0 and mrank(A) = 2 if t = 0.
If t = 0, then T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e2 hasrank 3.
If t 6= 0 then rank(T ) = 2 and
T = e2 ⊗ e1 ⊗ (e1 − t−1e2) + (e1 + t−1e2)⊗ (e1 + te2)⊗ e2
Harm Derksen Tensors, LRMC and Singular Values
Rank Minimization Problem (RM)
Problem (Rank Minimization)
Given A,B1, . . . ,Bs ∈ Matp,q(F), minimizerank(A + x1B1 + · · ·+ xsBs).
LRMC is a special case of RM where Bk = eik ,jk .
Theorem (D.)
The Tensor Rank problem can be reduced to RM and RM can bereduced to LRMC.
For example, suppose that T = (ti ,j ,k)2i ,j ,k=1 is a 2× 2× 2 tensor.
Harm Derksen Tensors, LRMC and Singular Values
Rank Minimization Problem (RM)
Problem (Rank Minimization)
Given A,B1, . . . ,Bs ∈ Matp,q(F), minimizerank(A + x1B1 + · · ·+ xsBs).
LRMC is a special case of RM where Bk = eik ,jk .
Theorem (D.)
The Tensor Rank problem can be reduced to RM and RM can bereduced to LRMC.
For example, suppose that T = (ti ,j ,k)2i ,j ,k=1 is a 2× 2× 2 tensor.
Harm Derksen Tensors, LRMC and Singular Values
t1,1,1 t1,2,1 a1,1 a1,2 a1,3 0 0 0 0 0 0 0 0 0 0 0 0 0 0t2,1,1 t2,2,1 a2,1 a2,2 a2,3 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 λ1,1 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 1 0 0 λ1,2 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 1 0 0 λ1,3 0 0 0 0 0 0 0 0 0 0 0
b1,1 b2,1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0b1,2 b2,2 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0b1,3 b2,3 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 t1,1,2 t1,2,2 a1,1 a1,2 a1,3 0 0 0 0 0 00 0 0 0 0 0 0 0 t2,1,2 t2,2,2 a2,1 a2,2 a2,3 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 1 0 0 λ2,1 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 1 0 0 λ2,2 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 1 0 0 λ2,3 0 0 00 0 0 0 0 0 0 0 b1,1 b2,1 0 0 0 1 0 0 0 0 00 0 0 0 0 0 0 0 b1,2 b2,2 0 0 0 0 1 0 0 0 00 0 0 0 0 0 0 0 b1,3 b2,3 0 0 0 0 0 1 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a1,1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a2,1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b1,1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b2,1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a1,2 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a2,2 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b1,2 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b2,2 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a1,30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a2,30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b1,30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b2,3
.
The smallest rank over all ai ,j , bi ,j , λi ,j (1 ≤ i ≤ 2, 1 ≤ j ≤ 3) is12 + rank(T ).
Harm Derksen Tensors, LRMC and Singular Values
Motivation: compressed sensing and convex relaxation
For x ∈ Rn, its sparsity is measured by ‖x‖0 = #{i | xi 6= 0}.
Problem
Given A ∈ Matm,n(R), b ∈ Rm, find a solution x ∈ Rn for Ax = bwith ‖x‖0 minimal (a sparsest solution).
But, ‖ · ‖0 is not convex and this optimization problem is difficult,
so instead we consider:
Problem (Basis Pursuit)
Given A ∈ Matm,n(R), b ∈ Rm, find a solution x ∈ Rn for Ax = bwith ‖x‖1 minimal.
Basis Pursuit can be solved by linear programming and is generallyfast. Under reasonable assumptions, Basis Pursuit also gives thesparsest solutions (Candes-Tao, Donoho).
Harm Derksen Tensors, LRMC and Singular Values
Motivation: compressed sensing and convex relaxation
For x ∈ Rn, its sparsity is measured by ‖x‖0 = #{i | xi 6= 0}.
Problem
Given A ∈ Matm,n(R), b ∈ Rm, find a solution x ∈ Rn for Ax = bwith ‖x‖0 minimal (a sparsest solution).
But, ‖ · ‖0 is not convex and this optimization problem is difficult,so instead we consider:
Problem (Basis Pursuit)
Given A ∈ Matm,n(R), b ∈ Rm, find a solution x ∈ Rn for Ax = bwith ‖x‖1 minimal.
Basis Pursuit can be solved by linear programming and is generallyfast. Under reasonable assumptions, Basis Pursuit also gives thesparsest solutions (Candes-Tao, Donoho).
Harm Derksen Tensors, LRMC and Singular Values
Nuclear Norm
V (i) Hilbert space. Instead of the CP model, we consider:
Problem (Convex Decomposition)
Given a tensor T , write T =∑r
i=1 vi for some r and some puretensors v1, . . . , vr such that
∑ri=1 ‖vi‖2 is minimal.
Finding a convex decomposition seems to be easier than finding aCP decomposition. Heuristically, we expect convex decompositionsto give a CP decomposition or at least a low rank decomposition.
Definition (Lim-Comon)
The nuclear norm ‖T‖? is the smallest value of∑r
i=1 ‖vi‖2 whereT =
∑ri=1 vi and v1, . . . , vr are pure tensors.
For some tensors we know the nuclear norm but not the rank.
Harm Derksen Tensors, LRMC and Singular Values
Nuclear Norm
V (i) Hilbert space. Instead of the CP model, we consider:
Problem (Convex Decomposition)
Given a tensor T , write T =∑r
i=1 vi for some r and some puretensors v1, . . . , vr such that
∑ri=1 ‖vi‖2 is minimal.
Finding a convex decomposition seems to be easier than finding aCP decomposition. Heuristically, we expect convex decompositionsto give a CP decomposition or at least a low rank decomposition.
Definition (Lim-Comon)
The nuclear norm ‖T‖? is the smallest value of∑r
i=1 ‖vi‖2 whereT =
∑ri=1 vi and v1, . . . , vr are pure tensors.
For some tensors we know the nuclear norm but not the rank.
Harm Derksen Tensors, LRMC and Singular Values
Spectral Norm
Definition
The spectral norm is defined by
‖T‖σ = max{|〈T , v〉| | v pure tensor with ‖v‖2 = 1}.
The spectral norm is dual to the nuclear norm, in particular
|〈T ,S〉| ≤ ‖T‖?‖S‖σ
for all tensors S ,T .
Harm Derksen Tensors, LRMC and Singular Values
Example: Determinant Tensor
Consider the tensor
Dn =∑σ∈Sn
sgn(σ)eσ(1) ⊗ eσ(2) ⊗ · · · ⊗ eσ(n) ∈ Cn ⊗ · · · ⊗ Cn.
Clearly rank(Dn) ≤ n!, actually( nbn/2c
)≤ rank(Dn) ≤ (56)bn/3cn!.
‖Dn‖σ = max{| det(v1v2 · · · vn)| | ‖v1‖2 = · · · = ‖vn‖2 = 1} = 1
by Hadamard’s inequality.
‖Dn‖? = ‖Dn‖?‖Dn‖σ ≥ 〈Dn,Dn〉 = n!, so
Theorem (D.)
‖Dn‖? = n!
Harm Derksen Tensors, LRMC and Singular Values
Example: Determinant Tensor
Consider the tensor
Dn =∑σ∈Sn
sgn(σ)eσ(1) ⊗ eσ(2) ⊗ · · · ⊗ eσ(n) ∈ Cn ⊗ · · · ⊗ Cn.
Clearly rank(Dn) ≤ n!, actually( nbn/2c
)≤ rank(Dn) ≤ (56)bn/3cn!.
‖Dn‖σ = max{| det(v1v2 · · · vn)| | ‖v1‖2 = · · · = ‖vn‖2 = 1} = 1
by Hadamard’s inequality.
‖Dn‖? = ‖Dn‖?‖Dn‖σ ≥ 〈Dn,Dn〉 = n!, so
Theorem (D.)
‖Dn‖? = n!
Harm Derksen Tensors, LRMC and Singular Values
Example: Permanent Tensor
Pn =∑
σ∈Sn eσ(1) ⊗ eσ(2) ⊗ · · · ⊗ eσ(n) ∈ Cn ⊗ · · · ⊗ Cn.
‖Pn‖σ = max{| perm(v1v2 · · · vn)| | ‖v1‖2 = · · · = ‖vn‖2 = 1}
Theorem (Carlen, Lieb and Moss, 2006)
max{perm(v1v2 · · · vn) | ‖v1‖2 = · · · = ‖vn‖2 = 1} = n!/nn/2
‖Pn‖? =nn/2
n!‖Pn‖?‖Pn‖σ ≥
nn/2
n!〈Pn,Pn〉 = nn/2.
Harm Derksen Tensors, LRMC and Singular Values
Example: Permanent Tensor
Pn =∑
σ∈Sn eσ(1) ⊗ eσ(2) ⊗ · · · ⊗ eσ(n) ∈ Cn ⊗ · · · ⊗ Cn.
‖Pn‖σ = max{| perm(v1v2 · · · vn)| | ‖v1‖2 = · · · = ‖vn‖2 = 1}
Theorem (Carlen, Lieb and Moss, 2006)
max{perm(v1v2 · · · vn) | ‖v1‖2 = · · · = ‖vn‖2 = 1} = n!/nn/2
‖Pn‖? =nn/2
n!‖Pn‖?‖Pn‖σ ≥
nn/2
n!〈Pn,Pn〉 = nn/2.
Harm Derksen Tensors, LRMC and Singular Values
Example: Permanent Tensor
Theorem (Glynn 2010)
Pn =1
2n−1
∑δ
(∏ni=1 δi
)(∑n
i=1 δiei )⊗ · · · ⊗ (∑n
i=1 δiei )
where δ runs over {1} × {−1, 1}n−1.
In particular,( nbn/2c
)≤ rank(Pn) ≤ 2n−1 and ‖Pn‖? ≤ nn/2, so
Theorem (D.)
‖Pn‖? = nn/2
Harm Derksen Tensors, LRMC and Singular Values
Example: Permanent Tensor
Theorem (Glynn 2010)
Pn =1
2n−1
∑δ
(∏ni=1 δi
)(∑n
i=1 δiei )⊗ · · · ⊗ (∑n
i=1 δiei )
where δ runs over {1} × {−1, 1}n−1.
In particular,( nbn/2c
)≤ rank(Pn) ≤ 2n−1 and ‖Pn‖? ≤ nn/2, so
Theorem (D.)
‖Pn‖? = nn/2
Harm Derksen Tensors, LRMC and Singular Values
t-Orthogonality
Definition
Pure tensors v1, v2, . . . , vr are t-orthogonal if
r∑i=1
|〈vi ,w〉|2/t ≤ 1
for every pure tensor w with ‖w‖2 = 1.
1-orthogonal ⇔ orthogonalIf t > s then t-orthogonal ⇒ s-orthogonal.
Theorem (D.)
If v1, . . . , vr ∈ V are t-orthogonal, then r ≤ dim(V )1/t .
Harm Derksen Tensors, LRMC and Singular Values
t-Orthogonality
Definition
Pure tensors v1, v2, . . . , vr are t-orthogonal if
r∑i=1
|〈vi ,w〉|2/t ≤ 1
for every pure tensor w with ‖w‖2 = 1.
1-orthogonal ⇔ orthogonalIf t > s then t-orthogonal ⇒ s-orthogonal.
Theorem (D.)
If v1, . . . , vr ∈ V are t-orthogonal, then r ≤ dim(V )1/t .
Harm Derksen Tensors, LRMC and Singular Values
t-Orthogonality
Definition
Pure tensors v1, v2, . . . , vr are t-orthogonal if
r∑i=1
|〈vi ,w〉|2/t ≤ 1
for every pure tensor w with ‖w‖2 = 1.
1-orthogonal ⇔ orthogonalIf t > s then t-orthogonal ⇒ s-orthogonal.
Theorem (D.)
If v1, . . . , vr ∈ V are t-orthogonal, then r ≤ dim(V )1/t .
Harm Derksen Tensors, LRMC and Singular Values
Horizontal and Vertical Tensor Product
Theorem (“horizontal tensor product”, D.)
If v1, . . . , vr are t-orthogonal, and w1, . . . ,wr are s-orthogonal,then v1 ⊗ w1, . . . , vr ⊗ wr are (s + t)-orthogonal.
If V = V (1) ⊗ · · · ⊗ V (d) and W = W (1) ⊗ · · · ⊗W (d), then
V �W := (V (1) ⊗W (1))⊗ · · · ⊗ (V (d) ⊗W (d)).
Theorem (“vertical tensor product”, D.)
If v1, v2, . . . , vr ∈ V and w1, . . . ,ws ∈W are t-orthogonal, then{vi � wj | 1 ≤ i ≤ r , 1 ≤ j ≤ s} are t-orthogonal.
Harm Derksen Tensors, LRMC and Singular Values
Horizontal and Vertical Tensor Product
Theorem (“horizontal tensor product”, D.)
If v1, . . . , vr are t-orthogonal, and w1, . . . ,wr are s-orthogonal,then v1 ⊗ w1, . . . , vr ⊗ wr are (s + t)-orthogonal.
If V = V (1) ⊗ · · · ⊗ V (d) and W = W (1) ⊗ · · · ⊗W (d), then
V �W := (V (1) ⊗W (1))⊗ · · · ⊗ (V (d) ⊗W (d)).
Theorem (“vertical tensor product”, D.)
If v1, v2, . . . , vr ∈ V and w1, . . . ,ws ∈W are t-orthogonal, then{vi � wj | 1 ≤ i ≤ r , 1 ≤ j ≤ s} are t-orthogonal.
Harm Derksen Tensors, LRMC and Singular Values
The Diagonal Singular Value Decomposition
Definition
Suppose that (?) : T =∑r
i=1 λivi such that λ1 ≥ · · · ≥ λr > 0and v1, . . . , vr are 2-orthogonal pure tensors of unit length, then(?) is called a diagonal singular value decomposition of T (DSVD).
If d = 2 (tensor product of 2 spaces) then the DSVD is the usualsingular value decomposition. For d > 2, the DSVD is differentfrom the Higher Order Singular Value Decomposition defined byDe Lathauer, De Moor, and Vandewalle. Not every tensor has aDSVD.
Harm Derksen Tensors, LRMC and Singular Values
The Diagonal Singular Value Decomposition
Theorem (D.)
If T has a DSVD then
‖T‖? =∑i
λi , ‖T‖2 =
√∑i
λ2i , ‖T‖σ = λ1
Theorem (D.)
If λ1 > λ2 > · · · > λr then the DSVD is unique.
Theorem (D.)
If v1, . . . , vr are t-orthogonal with t > 2, then the DSVD is unique.
Harm Derksen Tensors, LRMC and Singular Values
The Diagonal Singular Value Decomposition
Theorem (D.)
If T has a DSVD then
‖T‖? =∑i
λi , ‖T‖2 =
√∑i
λ2i , ‖T‖σ = λ1
Theorem (D.)
If λ1 > λ2 > · · · > λr then the DSVD is unique.
Theorem (D.)
If v1, . . . , vr are t-orthogonal with t > 2, then the DSVD is unique.
Harm Derksen Tensors, LRMC and Singular Values
Example: Matrix Multiplication Tensor
e1, . . . , en ∈ Cn are orthogonale1 ⊗ e1, . . . , en ⊗ en ∈ Cn ⊗ Cn are 2-orthogonal
e1 ⊗ e1 ⊗ 1, . . . , en ⊗ en ⊗ e1 ∈ Cn ⊗ Cn ⊗ C are 2-orthogonale1 ⊗ 1⊗ e1, . . . , en ⊗ 1⊗ en ∈ Cn ⊗ C⊗ Cn are 2-orthogonal1⊗ e1 ⊗ e1, . . . , 1⊗ en ⊗ en ∈ C⊗ Cn ⊗ Cn are 2-orthogonal
Using vertical tensor product, we get
{(ei ⊗ ej)⊗ (ej ⊗ ek)⊗ (ek ⊗ ei ) | 1 ≤ i , j , k ≤ n}
are 2-orthogonal.
Harm Derksen Tensors, LRMC and Singular Values
Example: Matrix Multiplication Tensor
e1, . . . , en ∈ Cn are orthogonale1 ⊗ e1, . . . , en ⊗ en ∈ Cn ⊗ Cn are 2-orthogonal
e1 ⊗ e1 ⊗ 1, . . . , en ⊗ en ⊗ e1 ∈ Cn ⊗ Cn ⊗ C are 2-orthogonale1 ⊗ 1⊗ e1, . . . , en ⊗ 1⊗ en ∈ Cn ⊗ C⊗ Cn are 2-orthogonal1⊗ e1 ⊗ e1, . . . , 1⊗ en ⊗ en ∈ C⊗ Cn ⊗ Cn are 2-orthogonal
Using vertical tensor product, we get
{(ei ⊗ ej)⊗ (ej ⊗ ek)⊗ (ek ⊗ ei ) | 1 ≤ i , j , k ≤ n}
are 2-orthogonal.
Harm Derksen Tensors, LRMC and Singular Values
Example: Matrix Multiplication Tensor
e1, . . . , en ∈ Cn are orthogonale1 ⊗ e1, . . . , en ⊗ en ∈ Cn ⊗ Cn are 2-orthogonal
e1 ⊗ e1 ⊗ 1, . . . , en ⊗ en ⊗ e1 ∈ Cn ⊗ Cn ⊗ C are 2-orthogonale1 ⊗ 1⊗ e1, . . . , en ⊗ 1⊗ en ∈ Cn ⊗ C⊗ Cn are 2-orthogonal1⊗ e1 ⊗ e1, . . . , 1⊗ en ⊗ en ∈ C⊗ Cn ⊗ Cn are 2-orthogonal
Using vertical tensor product, we get
{(ei ⊗ ej)⊗ (ej ⊗ ek)⊗ (ek ⊗ ei ) | 1 ≤ i , j , k ≤ n}
are 2-orthogonal.
Harm Derksen Tensors, LRMC and Singular Values
Example: Matrix Multiplication Tensor
Theorem (D.)
The matrix multiplication tensor
Tn =n∑
i ,j ,k=1
ei ,j ⊗ ej ,k ⊗ ek,i
is a DSVD.
The singular values of Tn are
1, 1, . . . , 1︸ ︷︷ ︸n3
In particular,
‖Tn‖? =n3∑i=1
1 = n3.
Harm Derksen Tensors, LRMC and Singular Values
Example: Discrete Fourier Transform
DefineFn =
∑1≤i,j,k≤n
i+j+k≡0 mod n
ei ⊗ ej ⊗ ek
This tensor is related to the multiplication of univariatepolynomials. Clearly rank(Fn) ≤ n2 and ‖Fn‖? ≤ n2.
Discrete Fourier Transform (DFT):
Fn =n∑
j=1
√n( 1√
n
∑ni=1 ζ
ijei )⊗ ( 1√n
∑ni=1 ζ
ijei )⊗ ( 1√n
∑ni=1 ζ
ijei ).
where ζ = eπi/n. This is the unique DSVD of Fn. So the singularvalues are
√n, . . . ,
√n (n times), rank(Fn) = n and ‖Fn‖? = n
√n.
Harm Derksen Tensors, LRMC and Singular Values
Example: Discrete Fourier Transform
DefineFn =
∑1≤i,j,k≤n
i+j+k≡0 mod n
ei ⊗ ej ⊗ ek
This tensor is related to the multiplication of univariatepolynomials. Clearly rank(Fn) ≤ n2 and ‖Fn‖? ≤ n2.
Discrete Fourier Transform (DFT):
Fn =n∑
j=1
√n( 1√
n
∑ni=1 ζ
ijei )⊗ ( 1√n
∑ni=1 ζ
ijei )⊗ ( 1√n
∑ni=1 ζ
ijei ).
where ζ = eπi/n. This is the unique DSVD of Fn. So the singularvalues are
√n, . . . ,
√n (n times), rank(Fn) = n and ‖Fn‖? = n
√n.
Harm Derksen Tensors, LRMC and Singular Values
Generalization: Group Algebra Multiplication Tensor
G is a group with n elements and CG ∼= Cn is the group algebra
TG =∑
g ,h∈Gg ⊗ h ⊗ h−1g−1.
DFT case corresponds to G = Z/nZ.
Theorem (D.)
TG has a DSVD and its singular values are√nd1, . . . ,
√nd1︸ ︷︷ ︸
d31
, . . . ,√
nds, . . . ,
√nds︸ ︷︷ ︸
d3s
where d1, d2, . . . , ds are the dimension of the irreduciblerepresentations of G .
Harm Derksen Tensors, LRMC and Singular Values
Generalization: Group Algebra Multiplication Tensor
G is a group with n elements and CG ∼= Cn is the group algebra
TG =∑
g ,h∈Gg ⊗ h ⊗ h−1g−1.
DFT case corresponds to G = Z/nZ.
Theorem (D.)
TG has a DSVD and its singular values are√nd1, . . . ,
√nd1︸ ︷︷ ︸
d31
, . . . ,√
nds, . . . ,
√nds︸ ︷︷ ︸
d3s
where d1, d2, . . . , ds are the dimension of the irreduciblerepresentations of G .
Harm Derksen Tensors, LRMC and Singular Values
Pn no DSVD
Suppose that Pn has singular values λ1, . . . , λr .
‖Pn‖? = nn/2 =∑r
i=1 λi
‖Pn‖σ = n!nn/2
= λ1
‖Pn‖22 = n! =∑r
i=1 λ2i
λ1
r∑i=1
λi = n! =r∑
i=1
λ2i
so λ1 = · · · = λr , and r = rλ1λ1
= ‖Dn‖?‖Dn‖σ = nn
n! .If n > 2 then r is not an integer!
Harm Derksen Tensors, LRMC and Singular Values
Dn no DSVD
Suppose that Dn has singular values λ1, . . . , λr . Thenλ1 = · · · = λr = 1 and r = n! (similar calculation).
v1, v2, . . . , vr
are 2-orthogonal, so r ≤ (dimV )1/2 = nn/2. So
n! = r ≤ nn/2
Not possible for n > 2.
Harm Derksen Tensors, LRMC and Singular Values
Slope decomposition
Definition
T = T1 + T2 + · · ·+ Ts is called a slope decomposition if〈Ti ,Tj〉 = ‖Ti‖?‖Tj‖σ for all i ≤ j and
‖T1‖?‖T1‖σ
<‖T2‖?‖T2‖σ
< · · · < ‖Tr‖?‖Tr‖σ
.
Slope decomposition is unique if it exists. If T has a DSVD, thenit also has a slope decomposition. But not every tensor has a slopedecomposition.
Harm Derksen Tensors, LRMC and Singular Values
Singular values for tensors with slope decomposition
Definition
Suppose that T = T1 + · · ·+ Ts is a slope decomposition,µi = ‖Ti‖σ, and λi = µi + µi+1 + · · ·+ µs for all i . Then T has
singular value λi with multiplicity ‖Ti‖?‖Ti‖σ −
‖Ti−1‖?‖Ti−1‖σ .
Dn has singular value 1 with multiplicity n!Pn has singular value n!
nn/2with multiplicity nn
n! .
Lemma
If T has slope decomposition and singular values λ1, . . . , λr withmultiplicities m1, . . . ,mr , then ‖T‖σ = λ1, ‖T‖? =
∑i miλi and
‖T‖22 =∑
i miλ2i .
Harm Derksen Tensors, LRMC and Singular Values
Singular values for tensors with slope decomposition
Definition
Suppose that T = T1 + · · ·+ Ts is a slope decomposition,µi = ‖Ti‖σ, and λi = µi + µi+1 + · · ·+ µs for all i . Then T has
singular value λi with multiplicity ‖Ti‖?‖Ti‖σ −
‖Ti−1‖?‖Ti−1‖σ .
Dn has singular value 1 with multiplicity n!Pn has singular value n!
nn/2with multiplicity nn
n! .
Lemma
If T has slope decomposition and singular values λ1, . . . , λr withmultiplicities m1, . . . ,mr , then ‖T‖σ = λ1, ‖T‖? =
∑i miλi and
‖T‖22 =∑
i miλ2i .
Harm Derksen Tensors, LRMC and Singular Values
Singular values for tensors with slope decomposition
Definition
Suppose that T = T1 + · · ·+ Ts is a slope decomposition,µi = ‖Ti‖σ, and λi = µi + µi+1 + · · ·+ µs for all i . Then T has
singular value λi with multiplicity ‖Ti‖?‖Ti‖σ −
‖Ti−1‖?‖Ti−1‖σ .
Dn has singular value 1 with multiplicity n!Pn has singular value n!
nn/2with multiplicity nn
n! .
Lemma
If T has slope decomposition and singular values λ1, . . . , λr withmultiplicities m1, . . . ,mr , then ‖T‖σ = λ1, ‖T‖? =
∑i miλi and
‖T‖22 =∑
i miλ2i .
Harm Derksen Tensors, LRMC and Singular Values
Generalizations
One can a singular spectrum for arbitrary tensors compatible withthe singular spectrum for tensors with slope decomposition, but:one may get continuous spectra or negative multiplicities.
One can define a slope decomposition and singular values wheneverwe have a vector space with two norms dual to each other.
Thank You!
Harm Derksen Tensors, LRMC and Singular Values
Generalizations
One can a singular spectrum for arbitrary tensors compatible withthe singular spectrum for tensors with slope decomposition, but:one may get continuous spectra or negative multiplicities.
One can define a slope decomposition and singular values wheneverwe have a vector space with two norms dual to each other.
Thank You!
Harm Derksen Tensors, LRMC and Singular Values
Generalizations
One can a singular spectrum for arbitrary tensors compatible withthe singular spectrum for tensors with slope decomposition, but:one may get continuous spectra or negative multiplicities.
One can define a slope decomposition and singular values wheneverwe have a vector space with two norms dual to each other.
Thank You!
Harm Derksen Tensors, LRMC and Singular Values