CMU SCS

Indexing and Mining Biological Images

Christos Faloutsos

CMU

CMU-05 2

CMU SCS

Outline

• Motivation - Problem Definition

• Proposed method

• Experiments

• Conclusions

CMU-05 3

CMU SCS

ViVo

• with Ambuj Singh, Vebjorn Ljosa, Arnab Bhattacharya (UCSB)

• Jia-Yu Tim Pan, HJ Yang (CMU)

CMU-05 4

CMU SCS

Detachment Development

Normal1 day after detachment

3 days after detachment

7 days after detachment

28 days after detachment

3 months after detachment

CMU-05 5

CMU SCS

Data and Problem

• (Data) Retinal images taken from cats• (Problem) What happens in retina after

detachment?– What tissues (regions) are involved? – How do they change over time?

• How will a program convey this info?• More than classification

“we want to learn what classifier learned”

CMU-05 6

CMU SCS

Why study retinal detachment

• Common damage to retina

• No effective treatment– Surgery or drugs (<100% recovery)

• Need to understand more about detachment development

skip

CMU-05 7

CMU SCS

Retina, its image, and the detachment

• retina

Layers of tissues stained by 3 antibodies (R,G,B)

skip

CMU-05 8

CMU SCS

Computer Scientist’s View of Retinal Detachment

normal detachment 7 days after

skip

CMU-05 9

CMU SCS

Detachment Development

Normal1 day after detachment

3 days after detachment

7 days after detachment

28 days after detachment

3 months after detachment

CMU-05 10

CMU SCS

How do the treatments do?

28 days afterreattachment surgery

6 days afterO2 treatment

CMU-05 11

CMU SCS

Outline

• Motivation - Problem Definition

• Proposed method

• Experiments

• Conclusions

CMU-05 12

CMU SCS

Main idea

• extract characteristic visual ‘words’

• Equivalent to characteristic keywords, in a collection of text documents

CMU-05 14

CMU SCS

Visual Vocabulary (ViVo) generation

Tile image

Extract color structure features

Independent component

analysis (ICA)

Visualvocabulary

CMU-05 15

CMU SCS

Proposed method: ViVo

• Textures are different. – Wavelet (Daubechies-4), MPEG-7 color

structure

• Local variation: partitioned into 64x64 “tiles”.

[f1, …, fm] “tile-vector”

skip

CMU-05 16

CMU SCS

ViVos

skip

CMU-05 17

CMU SCS

Outline

• Motivation - Problem Definition

• Proposed method

• Experiments

• Conclusions

CMU-05 18

CMU SCS

Evaluation of ViVo method

• how meaningful are the discovered ViVos?

• can they help in classification?

• generality?

• how else can they help biologists create hypotheses?

CMU-05 19

CMU SCS

Example ViVos

vivo Meaning Condition

Intact rod cell bodiesNormal Outer Nuclear Layer (ONL)

Intact rod cell bodies +

rhodopsin labellingONL at the beginning of detachment

Degenerate rod cell bodies +

rhodopsin +

hypertrophied Müller cells

Detached ONL

Intact rod cell bodies + rhodopsin + hypertrophied Müller cells

Detached ONL in oxygen treatment

CMU-05 20

CMU SCS

Goals:

• how meaningful are the discovered ViVos?

• can they help in classification?

• generality?

• how else can they help biologists create hypotheses?

CMU-05 21

CMU SCS

Quality of ViVo – by classification

N 1d 3d 7d 28d 28dr 6dO2 3m

N 7 2

1d 7

3d 12 1 1 1

7d 1 8 2

28d 1 23 2

28dr 1 21

6dO2 1 1 9

3m 5

Truth

Predicted

86% accuracy46 ViVos (90% energy)

CMU-05 22

CMU SCS

Goals:

• how meaningful are the discovered ViVos?

• can they help in classification?

• generality?

• how else can they help biologists create hypotheses?

CMU-05 23

CMU SCS

ViVos for protein images

CMU-05 24

CMU SCS

Protein images – MPEG7 CS

Giantin Hoechst LAMP2 NOP4 Tubulin

Giantin 30

Hoechst 30

LAMP2 50 9 1

NOP4 1 8 2

Tubulin 1 23

Truth

Predicted

84% accuracy4 ViVos (93% energy)1-NN classifier

CMU-05 25

CMU SCS

Evaluation of ViVo method

• how meaningful are the discovered ViVos?

• can they help in classification?

• generality?

• how else can they help biologists create hypotheses? ‘ViVo-annotation’!

CMU-05 26

CMU SCS

Automatic ViVo-annotation of images

• A tile represents a ViVo vk if the largest coefficient of the tile is along the kth basis vector

• A ViVo vk represents a class ci if the majority of its tiles are in that class

• For each image, the representative ViVos for the class are automatically highlighted

CMU-05 27

CMU SCS

Which tissue is significant on 7-day?

CMU-05 28

CMU SCS

6 days after O2 treatment

CMU-05 29

CMU SCS

28 days after surgery

CMU-05 31

CMU SCS

Conclusions:

• how meaningful are the discovered ViVos?

• can they help in classification?

• generality?

• how else can they help biologists create hypotheses?

CMU-05 32

CMU SCS

Outcome/status

• What are the key results so far?– ViVos: Automatic Visual Vocabulary

generation for biomedical image mining, Bhattacharya, Ljosa, Pan, Yang, Faloutsos, Singh (under review)

– Software – MATLAB code

• Tutorial in SIGMOD’05 (Murphy+Faloutsos)