A General Model for Relational Clustering

Bo Long and Zhongfei (Mark) ZhangComputer Science Dept./Watson SchoolSUNY Binghamton

Xiaoyun WuYahoo! Inc.

Philip S.YuIBM Watson Research Center

Multi-type Relational Data (MTRD) is Everywhere! Bibliometrics

Papers, authors, journals

Social networks People, institutions, friendship links

Biological data Genes, proteins, conditions

Corporate databases Customers, products, suppliers, shareholders

Papers

Authors

Key words

Challenges for Clustering! Data objects are not identically distributed:

Heterogeneous data objects (papers, authors). Data objects are not independent

Heterogeneous data objects are related to each other.

No IID assumption

Relational Data Flat Data?Paper ID word1 word2 …… author1 author2 …… ………… ………. …….

1 1 3 …… 1 0 …… ……… ……….. ……..

…… …… ……. ……. …… ……. …… ……… ………. ……..

Author ID Paper 1 Paper 2 …… ………… ………. …….

1 1 0 …… ……… ……….. ……..

…… …… ……. …… ……… ………. ……..

High dimensional and sparse data Data redundancy

Word ID Paper 1 Paper 2 …… ………… ………. …….

1 1 3 …… ……… ……….. ……..

…… …… ……. ……. ……… ………. ……..

Papers

Authors

Key words

Relational Data Flat Data? No interactions of hidden structures of

different types of data objects Difficult to discover the global community

structure.

users

Web pages

queries

A General Model: Collective Factorization on Related Matrices Formulate multi-type relational data as a set

of related matrices; cluster different types of objects

simultaneously by factorizing the related matrices simultaneously.

Make use of the interaction of hidden structures of different types of objects.

Data Representation Represent a MTRD set as a set of related matrices:

Relation matrix, R(ij), denotes the relations between ith type of objects and jth type of objects.

Feature matrix, F(i), denotes the feature values for ith type of objects.

Users

Movies Words

Authors

Papers

f

R(12) R (12)

R (23)

F(1)

Matrix Factorization

)()()( iii BCF

Exploring the hidden structure of the data matrix by its factorization:

.

Tjijiij CACR )( )()()()(

Feature basis

matrix

Cluster association

matrix

Model: Collective Factorization on Related Matrices (CFRM)

2)()(

1

)()(

1

2)()()()()(

,,

||||

||)(||min)()()(

ii

mji

iib

mji

Tjijiijija

ABC

BCFw

CACRwijiji

CFRM Model: Example

3

1

2

f

2)3()23()2()23()23(

2)2()12()1()12()12(

2)1()1()1()1(

||)(||

||)(||

||||

Ta

Ta

b

CACRw

CACRw

BCFw

Spectral Clustering Algorithms that cluster points using

eigenvectors of matrices derived from the data

Obtain data representation in the low-dimensional space that can be easily clustered

Traditional spectral clustering focuses on homogeneous data

Main Theorem:

2)()(

1

)()(

1

2)()()()()(

,,

||||

||)(||min)()()(

ii

mji

iib

mji

Tjijiijija

ABC

BCFw

CACRwijiji

mji

iTjjijTiija

mi

iTiiTiib

ICC

CCCRCtrw

CFFCtrwik

iTi

1

)()()()()()(

1

)()()()()(

)(

))()((

)()((max)()(

Algorithm Derivation: Iterative Updating

pj

pjTjjTjpjpa

mjp

TpjTjjpjpja

Tpppb

p

RCCRw

RCCRw

FFwM

1

)()()()()(

)()()()()(

)()()()(

))()((

))()((

))((

))((max )()()(

)( )()(

ppTp

ICCCMCtr

ppTp

where,

Spectral Relaxation

Apply real relaxation to C(p) to let it be an arbitrary orthornormal matrix.

By Ky-Fan Theorem, the optimal solution is given by the leading kp eigenvectors of M(p).

))((max )()()(

)( )()(

ppTp

ICCCMCtr

ppTp

Spectral Relational Clustering (SRC)

Spectral Relational Clustering: Example

Update C (1) as k1 leading eigenvectors of

Update C (2) as k2 leading eigenvectors of

Update C (3) as k3 leading eigenvectors of )23()2()2()23()23()3( )()( RCCRwM TT

a

3

1

2

TTa

TTa

RCCRw

RCCRwM

)()(

)()()23()3()3()23()23(

)12()1()1()12()12()2(

TTa RCCRwM )()( )12()2()2()12()12()1(

Advantages of Spectral Relational Clustering (SRC)

Simple as traditional spectral approaches Applicable to relational data with various

structures. Adaptive low dimension embedding Efficient: O(tmn2k). For sparse data, it is

reduced to O(tmzk) where z denotes the number of non-zero elements

Special case 1: k-means and spectral clustering Flat data: a special MTRD with only one feature

matrix F,

By the main theorem, k-means is equivalent to the trace maximization,

2

,||||min CBF

BC

max ( )T

k

T T

c c Itr C FF C

Special case 2: Bipartite Spectral Graph Partitioning (BSGP) Bipartite graph: a special case of MTRD with one

relation matrix R,

BSGP restricts the clusters of different types of objects to have one-to-one associations, i.e., diagonal constraints on A.

2)2()1(

)(

)(||)(||min

2)2()2(

1)1()1(

T

ICC

ICCCACR

kT

kT

Experiments Bi-type relational data:

Document-word data Tri-type relational data:

Category-document-word data. Comparing algorithms:

Normalized Cut (NC), Bipartite Spectral Graph Partitioning (BSGP), Mutual Reinforcement K-means (MRK) Consistent Bipartite Graph Co-partitioning

(CBGC).

Experimental Results on Bi-type Relational Data

Multi2 Multi3 Multi5 Multi8 Multi100

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8NMI Comparisons on Bi-type Relational Data

Norm

aliz

ed M

utu

al In

form

ation

SRC

NC

BSGP

Eigenvectors of a multi2 data set

-1 0 1-1

-0.5

0NC

u2

u 1

0 0.2 0.40

0.5

1BSGP

u2

-1 0 1-1

-0.5

0SRC

u2

u 1

0 5 100

1

2

Number of iterations

Obj

ectiv

e V

alue

Convergence

Experimental Results on Tri-type Relational Data

BRM TM1 TM2 TM30

0.2

0.4

0.6

0.8

1NMI Comparisons on Tri-type Relational Data

Norm

aliz

ed M

utu

al In

form

ation

SRC

MRKCBGC

Summary Collective Factorization on Related Matrices– a

general model for MTRD clustering. Spectral Relational Clustering– A novel spectral

approach Simple and applicable to relational data with various

structures. Adaptive low dimension embedding Efficient

Theoretic analysis and experiments demonstrate the effectiveness and the promise of the model and of the algorithm.