Smooth Non-negative Matrix Factorization

using Function Approximation

November 1, 2013

Lab. Seminar

Tatsuya YokotaRIKEN BSI

1

2Our interest

BIG DATA

[from IBM analytics website]

Visualization

Feature analysis

ClusteringClassification

With more precision !! With more fast !!

[3] Daniel D. Lee and H. Sebastian Seung (1999). "Learning the parts of

objects by non-negative matrix factorization". Nature 401 (6755): 788–791

There are many nonnegative real-valued data such as image,

text, and physical/chemical spectra.

NMF is a technique to decompose a nonnegative observed

matrix Y into two nonnegative matrices A and X.

Optimization criterion

3NMF: nonnegative matrix factorization

Update A Update X

Convergence !!

4Blind Source Separation by NMF

vectorization

Observed signal 1 Observed signal 2

Estimated Sources

NMF

5Clustering by NMF

・・・

・・・

・・・

・・・

・・・

Dataset Centroids Weight parameters

Visualization by PCA

We can obtain centroids and

its clusters at the same time!!

Penalty termFused penalty [Chen et al., 2006]

Gibbs regularization [Zdunek and Cichocki, 2007, 2008]

Function approximation (FA) [Zdunek, 2012]

Approximate a matrix A by function approximation

Φ is a set of smooth basis functions

W is a weight parameter matrix

6Smooth NMF

Function approximation

7Smooth basis function Φ

Approximate as smooth feature vector

×

For X

Smooth NMF (GRBF-NMF)

8Algorithm for Smooth NMF

Fixed matrix

optimize optimize

[Zdunek, 2012] Zdunek, Rafał. "Approximation of feature vectors in nonnegative matrix factorization

with Gaussian radial basis functions." Neural Information Processing. Springer Berlin Heidelberg, 2012.

NNLS problem (quickly-solvable)

For W

QP problem (slowly)

Convergence !!

Improvement algorithm very fast (fastGRBF)

Extension to tensor decomposition

9Our research

NNLS problem (quickly-solvable)

QP problem (slowly)

SLOWImprove by fast HALS

based algorithm

CP model Tucker model

gr

ar

cr

br

GA

C

BT

New Criterion

HALS algorithm (fast)

10FastGRBF-NMF

Update Update

For r = 1, …, R

Convergence !!

11Extension to Tensor

GA

C

BT

2-direcions

smoothing

GU

C

BT

GU

C

VT

1-direcion

smoothing

All-directions

smoothing

GU

W

VT

T

Dataset: Yale faces dataset

We used (65×51 pixels)×(10 images)

Salt and pepper noise added

12Result: Denoising

PSNR( , )

Denoising score

NMF

・・・

10 images

65x51

= 3315

pixels

Noise level 10 dB (10 images are used)

Compare (a)PSNR and (b)Computational time

13Result: Denoising

(a) PSNR

(b) Time

New

New

NewNew

Algorithm Multiplicative HALS GRBF fastGRBF

Ave. of times[s] 0.499 0.759 720.0 2.23

About 300 times faster

Toy problem

Evaluation criteria

14Result: Blind Source Separation

Linear

Mixing

System

Gaussian noise

(a) 5 smooth sources (b) 20 noisy observations (4.4 dB)

+ =NMF

SIR( , )

Source estimation

SIR( , )

Denoising

Result :

Toy problem

15Result: Blind Source Separation

New New

Original Estimator

Denoising Score Estimation Score

Result :

Far-infrared

Spectra

16Result: Blind Source Separation

NewNew

Original Gibbs Reg. fastGRBF

Denoising Score Estimation Score

CBCL faces:

we used (19×19 pixels) × 100 images

Added Gaussian noise

Matrix factorization model

Tenor factorization model

17Result: CBCL faces with Tensor model

・・・

100 images

19x19

= 361

pixels

Ordinary NMF

fastGRBF-NMF

100 images

19 pixels

19 pixels

GA

C

B TOrdinary Nonneg. Tucker Dcomp. (NTD)

Ordinary Nonneg. CP Dcomp. (NCPD)

GU

C

V T

fastGRBF-NTD-2way-smoothing

fastGRBF-NCPD-2way-smoothing

Result :

Matrix model

18Result: CBCL faces with matrix model

Noisy data 16.2 dB

3x3 smooth filter 22.8 dB

multipl. NMF 21.9 dB

fastGRBF-NMF 23.4 dB

Proposed methods

Ordinary methods

19Result: CBCL faces with tensor model

Result :

Tensor modelNTD 20.7 dB

NCPD 23.2 dB

fastGRBF-NTD 23.2 dB

fastGRBF-NCPD 23.3 dB

Proposed methods

Ordinary methods

20Result: 3d-tensor data denoising

7.21 dB

NCPD

(19.8 dB)

NTD

(13.5 dB)

fastGRBF-NCPD

(26.8 dB)

fastGRBF-NTD-

(23.9 dB)

Gaussian

noise

GA

C

B T

GU

W

V TT

Ordinary model

fastGRBF-NTD/NCPD-3way-smoothing

Summary

We proposed new fast algorithm for GRBF-NMF

It succeeded about 300 faster computational time

We extended its model to tensor decomposition model

Their method performed good results

Future work for CIFA

Common smoothness and individual sparseness

21Future work

Y1 ≒ AC A1 X1

Y2 ≒ AC A2 X2

Common smooth factor Individual sparse factor