Mining Biological Data

Jiong Yang, Ph. D. Visiting Assistant ProfessorUIUC jioyang@cs.uiuc.edu

Data is Everywhere

Data Mining is a Powerful Tool

Computational Biology E-Commerce Intrusion Detection Multimedia Processing Unstructured Data . . .

Data

Data MiningKnowledge

Biological Data Bio-informatics have become one of the most

important applications in data mining.DNA sequencesProtein sequencesProtein foldingMicroarray data……

Outline

Approximate sequential pattern mining

Coherent cluster: clustering by pattern similarity in a large data set

Frequent Patterns Model

A set of sequences of symbols. a1,a2,a4 a2,a3,a5 a1,a4,a5,a6,a7

If a pattern occurs more than a certain number of times, then this pattern is considered important.

a1,a4 Widely studied

Frequent itemset mining: Agarwal and Srikant (IBM Almaden) FP growth: Han (UIUC) Stream data: Motwani (Stanford) …

Apriori Property Widely used in data mining field It holds for the support metrics All patterns form a lattice.

(a, b, d) is a super-pattern of (a, d) and it is a sub-pattern of (a, b, c, d).

Support metric defines a partial order on the lattice. Support(a, b, d) <= min{Support(b, d) , Support(a, d) ,

Support(a, b) }Level-wise search algorithm can be used

Shortcomings Require exact match and fail to recognize

possible substitution among symbolsProtein may mutate without change of its

functionality.A sensor may make some mistakesDifferent web pages may have similar contents.A word may have many synonyms.

How can the symbol substitution be modeled

Compatibility Matrix

d1 d2 d5d3 d4

d1

d2

d5

d3

d4

observedtrue

0.90.05

0.1 0

0.75

0.80.7

0

0.050.05

0.05

0.1 0.1

0.10.10.15

0.850.15

00

000 0 0

Compatibility matrix of 5 symbols

Compatibility Matrix The compatibility matrix serves as a bridge between

the observation and the underlying substance. Each observed symbol is interpreted as an occurrence of a

set of symbols with various probabilities. An observed symbol combination is treated as an

occurrence of a set of patterns with various degrees. Obtain the compatibility matrix through

empirical study domain expert

Match A new metric, match, is then proposed to

quantify the importance of a pattern.The match of a pattern P in a subsequence s (with

the same length) is defined as the conditional probability Prob(P| s).

The match of a pattern P in a sequence S is defined as the maximal match of P in every distinct subsequence in S.

A dynamic programming technique is used to compute the match of P in a sequence S

Match M(d1d2…di, S1S2…Sj) is the maximum of M(d1s2…di, S1S2…Sj-1)

and M(d1d2…di-1,S1S2…Sj-1) x C(di, Sj)

The match of a pattern P in a set of sequence is defined as the sum of the pattern P with each sequence.

A pattern is called a frequent pattern if its match exceeds a user-specified threshold min_match.

SpSp

max d1 d3

d4 d1d1d2

S

p 0.9 0.9 0.9 0.90.045 0.090.09

Challenges Previous work focuses on short patterns. Long patterns require a large number of

scans through the input sequence.Expensive I/O cost

Performance vs. Accuracy Probabilistic Approach

Chernoff Bound Let X be a random variable whose range is R. Suppose

that we have n independent observations of X and the observed mean is . The Chernoff bound states that, with probability (1- ), the true mean of X is at least - , where

With probability (1- ), the true value of X is at most + .

nR

2)/1ln(2

Approach Three-stage approach to mine patterns with length l:

Finding Match of Individual Symbols and Take a Sample set of sequences

Pattern Discovery on Samples Ambiguous Patterns Determination

Pattern Discovery on Samples Sample size: depending on memory size Based on the samples, three types of patterns are

determined.

Approach Frequent pattern if match is greater than (min_match

+) Ambiguous pattern if match is between (min_match - )

and (min_match + ). Infrequent pattern otherwise;

Ambiguous Patterns Ambiguous Patterns

Too manyBorder collapse

We have the negative and positive borders of significant patterns.

Our goal is to collapse the border as fast as possible.

Ambiguous Patterns

(d1)

(d1,d2) (d1,d3) (d1,d4) (d1,d5)

(d1,d2,d3) (d1,d2,d4) (d1,d2,d5) (d1,d3,d4) (d1,d3,d5)

(d1,d2,d3,d4) (d1,d2,d3,d5) (d1,d2,d4,d5) (d1,d3,d4,d5)

(d1,d2,d3,d4,d5)

(d1,d4,d5)

Ambiguous Patterns

(d1)

(d1,d2,) (d1,d3) (d1,d4) (d1,d5)

(d1,d2,d3) (d1,d2,d4) (d1,d2,d5) (d1,d3,d4) (d1,d3,d5) (d1,d4,d5)

(d1,d2,d3,d4) (d1,d2,d3,d5) (d1,d2,d4,d5) (d1,d3,d4,d5)

(d1,d2,d3,d4,d5)

frequent

infrequent

Effects of 1-

0

20

40

60

80

100

120

140

0.85 0.9 0.95 1

confidence

num

ber

of a

mbi

guou

s pa

tter

ns (K

)

1.00E-06

1.00E-05

1.00E-04

1.00E-03

1.00E-02

1.00E-01

1.00E+00

0.85 0.9 0.95 1

confidence

erro

r ra

te

With BorderCollapse

Without BorderCollapse

Approximate Pattern Mining Reference:

Mining long sequential patterns in a noisy environment, Proceeding of ACM SIGMOD International Conference on Management of Data (SIGMOD), pp. 406-417, 2002.

Other WorkPeriodic Patterns (KDD2000, ICDM2001)Statistically significant Patterns (KDD2001, ICDM

2002)

Outline Approximate sequential pattern mining

Coherent cluster: clustering by pattern similarity in a large data set

Coherent Cluster In many applications, data can be of very high

dimensionality. Gene expression data

Dozens to hundreds conditions/samples Customer evaluation

Thousands or more merchants

Objective: discover peer groups

dij

attributes

obje

cts

oi

aj

o1...

..

.

a1 . . . . . .

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16139 69 0 69 139 139 139 139 69 0 0 69 110 0 69 0 0

0 69 69 69 110 110 110 110 69 0 69 69 110 110 69 0 69139 110 0 69 69 110 139 139 139 0 69 69 139 69 69 0 0139 110 0 69 110 110 110 139 110 0 69 110 139 69 69 0 69208 179 110 69 110 110 110 161 161 0 69 69 110 0 69 0 69

0 0 0 69 69 139 161 179 139 0 69 0 110 0 69 0 690 0 0 0 110 110 110 69 110 0 0 0 69 0 69 0 69

179 161 69 69 69 110 69 110 110 0 0 69 0 0 69 0 6969 110 69 110 110 161 110 69 139 69 69 110 110 139 110 69 11069 0 69 69 110 139 110 0 0 0 69 69 110 69 69 0 0

139 161 110 110 139 179 139 110 139 69 69 69 110 110 110 69 69179 179 161 139 161 195 161 161 161 110 161 161 139 139 161 110 110179 240 161 195 195 256 220 208 240 139 195 195 195 161 195 161 110161 161 69 110 139 161 139 110 161 69 110 139 69 69 110 69 69208 283 240 248 264 304 283 283 283 195 220 240 240 240 248 195 208161 195 110 139 195 248 179 161 220 110 179 195 161 179 208 110 110139 161 139 161 139 179 161 139 69 69 139 69 69 179 179 110 69304 326 304 322 326 350 340 376 318 248 314 283 314 318 326 264 26469 69 0 69 110 110 69 0 69 0 69 69 139 69 69 0 0

283 208 220 277 289 326 289 289 248 220 271 240 271 294 277 230 208337 383 383 413 414 403 381 393 343 350 369 358 347 358 356 314 289161 161 220 195 161 195 161 110 110 110 195 179 179 69 139 110 110208 195 220 161 139 161 161 110 139 110 195 195 195 69 161 139 139248 230 330 300 277 240 240 179 195 220 277 289 240 240 220 161 161264 300 289 264 277 277 289 277 300 248 283 271 294 256 264 271 283230 240 289 264 240 256 220 208 220 248 271 256 256 240 220 179 208439 442 464 456 451 422 417 403 432 510 438 442 450 462 419 476 476256 230 208 240 230 248 240 283 248 220 230 230 220 240 248 220 240374 322 322 300 330 356 361 333 369 376 369 374 369 343 361 393 399139 195 161 139 161 139 161 139 179 110 110 139 139 139 110 161 161230 277 256 248 264 271 248 240 256 220 230 230 256 208 208 240 230494 470 498 488 477 460 466 484 449 532 485 473 464 487 477 492 484326 248 240 289 300 294 289 264 277 248 283 283 277 283 277 271 283179 139 110 69 69 110 69 69 69 69 69 69 69 69 110 69 69326 411 397 383 371 347 314 277 330 264 289 283 304 264 264 340 343161 220 220 220 208 208 161 161 208 179 195 179 179 161 139 161 139220 271 248 230 240 248 240 179 248 208 208 220 230 220 179 230 230220 271 230 208 161 195 161 161 195 161 208 195 220 161 179 195 220179 195 110 161 139 179 161 179 161 69 110 139 139 139 161 139 161

17 conditions40 genes

Coherent Cluster

0

100

200

300

400

500

600

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

condition

expr

essi

on le

vel

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16139 69 0 69 139 139 139 139 69 0 0 69 110 0 69 0 0

0 69 69 69 110 110 110 110 69 0 69 69 110 110 69 0 69139 110 0 69 69 110 139 139 139 0 69 69 139 69 69 0 0139 110 0 69 110 110 110 139 110 0 69 110 139 69 69 0 69208 179 110 69 110 110 110 161 161 0 69 69 110 0 69 0 69

0 0 0 69 69 139 161 179 139 0 69 0 110 0 69 0 690 0 0 0 110 110 110 69 110 0 0 0 69 0 69 0 69

179 161 69 69 69 110 69 110 110 0 0 69 0 0 69 0 6969 110 69 110 110 161 110 69 139 69 69 110 110 139 110 69 11069 0 69 69 110 139 110 0 0 0 69 69 110 69 69 0 0

139 161 110 110 139 179 139 110 139 69 69 69 110 110 110 69 69179 179 161 139 161 195 161 161 161 110 161 161 139 139 161 110 110179 240 161 195 195 256 220 208 240 139 195 195 195 161 195 161 110161 161 69 110 139 161 139 110 161 69 110 139 69 69 110 69 69208 283 240 248 264 304 283 283 283 195 220 240 240 240 248 195 208161 195 110 139 195 248 179 161 220 110 179 195 161 179 208 110 110139 161 139 161 139 179 161 139 69 69 139 69 69 179 179 110 69304 326 304 322 326 350 340 376 318 248 314 283 314 318 326 264 26469 69 0 69 110 110 69 0 69 0 69 69 139 69 69 0 0

283 208 220 277 289 326 289 289 248 220 271 240 271 294 277 230 208337 383 383 413 414 403 381 393 343 350 369 358 347 358 356 314 289161 161 220 195 161 195 161 110 110 110 195 179 179 69 139 110 110208 195 220 161 139 161 161 110 139 110 195 195 195 69 161 139 139248 230 330 300 277 240 240 179 195 220 277 289 240 240 220 161 161264 300 289 264 277 277 289 277 300 248 283 271 294 256 264 271 283230 240 289 264 240 256 220 208 220 248 271 256 256 240 220 179 208439 442 464 456 451 422 417 403 432 510 438 442 450 462 419 476 476256 230 208 240 230 248 240 283 248 220 230 230 220 240 248 220 240374 322 322 300 330 356 361 333 369 376 369 374 369 343 361 393 399139 195 161 139 161 139 161 139 179 110 110 139 139 139 110 161 161230 277 256 248 264 271 248 240 256 220 230 230 256 208 208 240 230494 470 498 488 477 460 466 484 449 532 485 473 464 487 477 492 484326 248 240 289 300 294 289 264 277 248 283 283 277 283 277 271 283179 139 110 69 69 110 69 69 69 69 69 69 69 69 110 69 69326 411 397 383 371 347 314 277 330 264 289 283 304 264 264 340 343161 220 220 220 208 208 161 161 208 179 195 179 179 161 139 161 139220 271 248 230 240 248 240 179 248 208 208 220 230 220 179 230 230220 271 230 208 161 195 161 161 195 161 208 195 220 161 179 195 220179 195 110 161 139 179 161 179 161 69 110 139 139 139 161 139 161

40 genes

Coherent Cluster

0

100

200

300

400

500

600

3 5 9 14 15

condition

expr

essi

on le

vel

YBL069WYBL097WYBR064WYBR065CYBR114WYCL013WYDR149CYDR461WYDR526CYHR061CYIL092WYIR043CYJL010CYJL023CYJL033WYJL076WYJR162CYKL068WYKL134CYLR219W

Co-regulated genes

Coherent Cluster• Observations:

1. If mapped to points in high dimensional space, they may not be close to each other.

• Bias exists universally.2. Only a subset of objects and a subset of attributes

may participate.3. Need to accommodate some degree of noise.

• Solution: subspace cluster, bicluster, coherent cluster

Subspace cluster CLICK: Argawal et al IBM Almaden

Find a subset of dimensions and a subset of objects such that the distance between the objects on the subset of dimensions is close.

The clusters may overlap Proclus: Aggawal et al IBM T. J. Watson

Do not allow overlap

Bicluster Developed in 2000 by Cheung and Church Using mean squared error residual After discovering one cluster, replace the

cluster with random data and find another Not efficient and not accurate

Coherent Cluster Coherent cluster

Subspace clustering Measure distance on mutual bias

pair-wise disparity For a 22 (sub)matrix consisting of objects {x, y} and

attributes {a, b}

)()( ybxbyaxa

ybya

xbxa

dddd

dddd

D

x

ya b

dxa

dya

dxb

dyb

x

ya battribute

mutual biasof attribute a

mutual biasof attribute b

Coherent ClusterA 22 (sub)matrix is a -coherent cluster if its D

value is less than or equal to .An mn matrix X is a -coherent cluster if every

22 submatrix of X is -coherent cluster. A -coherent cluster is a maximum -coherent cluster

if it is not a submatrix of any other -coherent cluster.Objective: given a data matrix and a threshold ,

find all maximum -coherent clusters.

Coherent Cluster Challenges:

Finding subspace clustering based on distance itself is already a difficult task due to the curse of dimensionality.

The (sub)set of objects and the (sub)set of attributes that form a cluster are unknown in advance and may not be adjacent to each other in the data matrix.

The actual values of the objects in a coherent cluster may be far apart from each other.

Each object or attribute in a coherent cluster may bear some relative bias (that are unknown in advance) and such bias may be local to the coherent cluster.

Coherent ClusterCompute the maximum coherent

attribute sets for each pair of objects

Construct the lexicographical tree

Post-order traverse the tree to find maximum coherent clusters

Compute the maximum coherent object sets for each pair of attributes

Two way pruning

Coherent Cluster Observation: Given a pair of objects {o1, o2} and a

(sub)set of attributes {a1, a2, …, ak}, the 2k submatrix is a -coherent cluster iff, for every attribute ai, the mutual bias (do1ai – do2ai) does not differ from each other by more than .

a1 a2 a3 a4 a51

3

5

7

3 2 3.5 2 2.5

o1

o2

[2, 3.5]

If = 1.5,then {a1,a2,a3,a4,a5} is acoherent attribute set (CAS)of (o1,o2).

a1 a2 a3 a4 a51

3

5

7

3 2 3.5 2 2.5

r1

r2

Coherent Cluster Strategy: find the maximum coherent attribute

sets for each pair of objects with respect to the given threshold .

= 1

3

5

7

r1

r2

a2

2a3

3.5a4

2a5

2.5a1

3

1

The maximum coherent attribute sets define the search space for maximum coherent clusters.

Two Way Pruninga0 a1 a2

o0 1 4 2o1 2 5 5o2 3 6 5o3 4 200 7o4 300 7 6

(o0,o2) →(a0,a1,a2)(o1,o2) →(a0,a1,a2)

(a0,a1) →(o0,o1,o2)(a0,a2) →(o1,o2,o3)(a1,a2) →(o1,o2,o4)(a1,a2) →(o0,o2,o4)

(o0,o2) →(a0,a1,a2)(o1,o2) →(a0,a1,a2)

(a0,a1) →(o0,o1,o2)(a0,a2) →(o1,o2,o3)(a1,a2) →(o1,o2,o4)(a1,a2) →(o0,o2,o4)

delta=1 nc =3 nr = 3

MCAS MCOS

Coherent Cluster High expressive power

The coherent cluster can capture many interesting and meaningful patterns overlooked by previous clustering methods.

Efficient and highly scalable Wide applications

Gene expression analysis Collaborative filtering

0

2000

4000

6000

8000

10000

12000

10 20 50 100 200 500

number of conditionsav

erag

e re

spon

se time (sec

)

traditionalclustering

coherent clustering

Coherent Cluster References:

Delta-cluster: capturing subspace correlation in a large data set, Proceedings of the 18th IEEE International Conference on Data Engineering (ICDE), pp. 517-528, 2002.

Clustering by pattern similarity in large data sets, Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD), pp. 394-405, 2002.

Enhanced biclustering on expression data, Proceedings of the IEEE bio-informatics and bioengineering (BIBE), 2003.

Other Work STING (VLDB1997) STING+ (ICDE1999, TKDE 2000) CLUSEQ (CSB2002, ICDE2003) Cluster Streams (ICDE2003)

Remarks Similarity measure

Powerful in capturing high order statistics and dependencies

Efficient in computation Robust to noise

Clustering algorithm High accuracy High adaptability High scalability High reliability