on the extension of trace norm to tensors

8/8/2019 On the extension of trace norm to tensors

1/16

On the extension of trace norm to tensors

Ryota Tomioka1, Kohei Hayashi2, Hisashi Kashima1

1The University of Tokyo2

Nara Institute of Science and Technology

2010/12/10

NIPS2010 Workshop: Tensors, Kernels, and Machine Learning


2/16

Convex low-ranktensorcompletion

Sensors

Time

Features

Factors

(loadings)

Core

(interactions)

I1

I2I3

I1

I2

I3

r1

r2

r3r1

r2r3

Tucker decomposition


3/16

Conventional formulation (nonconvex)

mode-k productobservation

minimizeC,U1,U2,U3

(Y C 1 U1 2 U2 3 U3) 2

F + regularization.

minimizeX

(YX)

2

F

s.t. rank

(X) (

r1, r2, r3

).

Alternate minimization

Have to fix the rank beforehand


4/16

Our approach

MatrixEstimation of

low-rank matrix

(hard)

Trace norm

minimization

(tractable)[Fazel, Hindi, Boyd 01]

Tensor

Estimation oflow-rank tensor

(hard)Rank defined in the sense of

Tucker decomposition

Extended

trace norm

minimization

(tractable)

Generalization


5/16

Trace norm regularization (for matrices)

X RRC, m = min(R, C)

Linear sum of

singular-values

Roughly speaking, L1 regulariza6on on the singularvalues.

Stronger regulariza6on > more zero singularvalues >

low rank.

Not obvious for tensors (no singularvalues for tensors)

Xtr =

m

j=1

j(X)


6/16

Mode-k unfolding (matricization)

I1

I2 I3

I1

I2 I2 I2

I3

Mode-1 unfolding

I1

I2 I3

I2

I3 I3 I3

I1

Mode-2 unfolding

X(1)

X(2)


7/16

Elementary facts about Tucker decomposition

X = C 1 U1 2 U2 3 U3

r1

r2r3

C

I1r1

U1 I2

r2U2

I3

r3

U3

Mode-1 unfolding

Mode-2 unfolding

Mode-3 unfolding

X(1) = U1C(1)(U3 U2)

X(2) =

U2C

(2)(U

1U

3)

X(3) = U3C(3)(U2 U1)

rankr1

rankr2

rankr3


8/16

What it means

We can use the trace norm of an unfolding of a

tensorXto learn lowrankX.

Matricization

Tensorization


9/16

Approach 1: As a matrix

Pick a mode k, and hope that the tensor to be

learned is low rank in mode k.

minimizeXR

I1IK

12

(Y X) 2F + X(k),

Pro: Basically a matrix problem Theoretical guarantee (Candes & Recht 09)

Con: Have to be lucky to pick the right mode.


10/16

Approach 2: Constrained optimization

Constrain so that each unfolding ofXis

simultaneously low rank.

minimizeXR

I1IK

12

(Y X) 2F +K

k=1

kX(k).

k: tuning parameter usually set to 1.

Pro: Jointly regularize every modeCon: Strong constraint

See also Marco Signoretto et al.,10


11/16

Approach 3: Mixture of low-rank tensors

Each mixture componentZk is regularized to be

lowrank only in modek.

minimizeZ1,...,ZK

1

2

Y

Kk=1

Zk

2

F

+K

k=1

kZk(k),

Pro: Each Zk takes care of each modeCon: Sum is not low-rank


12/16

Numerical experiment

True tensor: Size 50x50x20, rank 7x8x9. No noise (=0).

Random train/test split.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

104

10

2

100

102

Fraction of observed elements

Genera

lizationerror

As a Matrix (mode 1)

As a Matrix (mode 2)As a Matrix (mode 3)

Constraint

Mixture

Tucker (large)

Tucker (exact)Optimization tolerance

Tucker=EM

algo

(nonconvex)


13/16

Computation time

Convex formula6on is also fast

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

10

20

30

40

50

Fraction of observed elements

Computationtime(s)

As a Matrix

ConstraintMixture

Tucker (large)

Tucker (exact)


14/16

Phase transition behaviour

0 20 40 60 80

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

[10 12 13]

[10 12 9][16 6 8]

[17 13 15]

[20 20 5]

[20 3 2]

[30 12 10]

[32 3 4]

[40 20 6]

[40 3 2]

[40 9 7]

[4 42 3]

[5 4 3]

[7 8 15]

[7 8 9][8 5 6]

Sum of true ranks

Fraction

required

forperfectrecon

struction

Sum of true ranks= min(r1,r2r3)+ min(r2,r3r1)+ min(r3,r1r2)


15/16

Summary

Lowrank tensor comple6on can be computed in a convex

op6miza6on problem using the trace norm of the unfoldings.

No need to specify the rank beforehand.

Convex formula6on is more accurate and faster thanconven6onal EMbased Tucker decomposi6on.

Curious phase transi6on found compressivesensing

type analysis is an ongoing work.

Technical report: Arxiv:1010.0789 (including op6miza6on)

Code:

h]p://www.ibis.t.utokyo.ac.jp/RyotaTomioka/So_wares/Tensor


16/16

Acknowledgment

This work was supported in part by MEXT KAKENHI

22700138, 22700289, and NTT Communica6on

Science Laboratories.

on the extension of trace norm to tensors

Documents