Outline Content-Based Image Retrieval

Query-by-Example Query-by-Feature Feature Vector

CBIR and CBR

Content-based Image Retrieval (CBIR) como exemplo de Content-based Retrieval (CBR)

concentra em low-level features. Principais idéias de CBIR:

Representar uma imagem como um conjunto de feature descriptors.

Definir medidas de similaridade dos descritores Quando um usuário especificar uma query, o sistema

retorna imagens, que são ordenadas por similaridade.

CBIR Architecture

FeatureExtraction

User Interface

SimilarityMetric

DatabaseImage data

Image Representation

Imagedata

ImageBrowsing

DatabaseCreation

QueryComparison

query

4

Image Retrieval

FeatureExtraction

Feature Vectors

FeatureExtraction

Compare

Select

Database Images

Imag

e D

atab

ase

Met

adat

abas

e

Query Image

Query Result

CBIR de Butterflies Permitir non-expert users encontrar

algumas espécies de butterflies usando informações de aparência de butterflies

Aparência: Color, Texture, Shape

Problemas

Como podemos descrever uma butterfly?

Como podemos comunicar nossa descrição para uma máquina?

Problemas

Usuários diferentes têm percepções diferentes.

Usuários podem não se lembrar claramente a aparêcia de uma butterfly.

Usuários normalmente não têm expertise para descrever butterflies.

Usuários normalmente não têm paciência para fazer o browse num grande conjunto resultado.

Soluções

Usar um processo de consulta interativo e direcionado ao usuário: QBF/QBE query process Query By Features e Query By Example

Fuzzy feature description para cada butterfly Uma “What You See Is What You Get” query

interface Um conjunto representativo de coleção

butterflies

QBF/QBE query process (1)

QBF query: A QBF query is to choose some features of

butterflies and expect that the system returns all butterflies with those features.

Features of butterflies: Dominant color, texture pattern, shape.

QBE query: A QBE query is to point an image and expect

that the system returns all butterflies similar to that.

QBF/QBE query process (2) Properties of QBF:

Rough search When to use:

The first query and when users want to enlarge the view in the search space

Properties of QBE: Fine search When to use:

Usually the last query and when users want to see the neighbors of the query one in the search space.

QBF/QBE query process (3)

Result page: (Each result page should contain two parts)

Result Images: These are the butterfly images satisfy the query

conditions. Users can invoke QBE queries from these images.

Related Features: These are the features related to the previous query

conditions. Users can invoke QBF queries from these features.

Feature Description (1)

Feature Description for a butterfly: Like metadata which describe the appearance

of this butterfly. This makes QBF queries possible. Feature Description consists of some feature

descriptors. Feature descriptor:

A ( “feature value” , “match level” ) pair.

Feature Description (2)

FeatureType Feature Value

Degree ofMatch

mixed_with_black_and_orange 52/57 orange_yellow 12/42Color orange_red 3/38 many_spots 58/62 fore_half_different_color 27/33 horizontal_bands 41/60

Texture

edge_with_different_color 10/74Shape wave 98/110

Feature Description (3)

gray

white

mixed_with_black_and_white

mixed_with_black_and_yellow

mixed_with_black_and_orange

mixed_with_black_and_blue

mixed_with_black_and_red

mixed_with_wood_and_white

mixed_with_many_colors

Color

Figure Feature Value black

brown

bister

orange_red

orange_yellow

yellow

green

blue

purple

Feature Description (4)

Figure Feature Value

vertical_bands

horizontal_bands

many_bands

two_lines

many_lines

Obvious_vein

grids

eyes

few_spot

some_spots

many_spots

color_blocks

grainy

edge_with_different_color

starlike

fore_half_different_color

Texture

Feature Description (5)

Figure Feature Value

swallowlike

swallowtail

broken

wave

like_leaf

like_moth

with_little_tails

Shape

Feature Description (6)

QBF query: Single feature query:

Result images: images with its corresponding degree of match > 0.

Ranked by: degree of match. We call this ranked sequence “Feature

sequence.” Multiple features query:

Merge the corresponding feature sequences.

Result Presentation For QBF query:

Property: rough search Presentation: representative butterflies only

For QBE query: Property: fine search Presentation:

For very similar images: present them all For less similar images: representative ones

Feature Vector Indexing Goal:

To make search efficiently. Problems of Indexing in CBIR:

Dimension of feature space is very high. Index structure should support Euclidean

and non-Euclidean similarity measures. Solution:

Dimension reduction: KLT, DCT, DWT. Similarity indexing: R*-tree, SS-tree, SR-

tree.

Semi-Automatic Feature Extraction

Segmentation: Background segmentation Butterfly object segmentation

Feature extraction: Color: color histogram Texture: manual annotation Shape: manual annotation

Classic CBIR with Color Feature

Most of the CBR systems rely on the notion of color, this may differ: Dominant color Scalable color based on color histograms

(local for one region, global for the whole image)

Color Structure Descriptor (incoporates the spatial structure)

What color is the apple ? We are so visual !!!!

I’d say it isBright Red

I think it is“Crimson”

It isRed!

I really couldn’t tell you(I am color blind)

Color Histogram: Representation

A list of Color-Percentage pairs: Describe the colors and its percentages in an

image.

Nj

jjjjjc NjandPPColorValueIP(If1

1 ,1,10,),

Color Quantization Indexed Colors A jpg Image with 256-color components in each

RGB channel 256 x 256 x 256 colors in total → n groups, e.g,

in 256 groups, that makes a reduction 256x256, I.e., that each group takes 256 colors to count.

Similarity Measures - Overview

Minkowski Similarity Distance L1 : r = 1

Distance L2 : r = 2

Quadratic Similarity

Intersection Similarity

(Swain et Ballard 1991)

)hh(A)hh()h,h(d tqT

tqtq

1M

0kt

1M

0ktq

t,q

]k[h

])k[h],k[hmin(

d

1r )]k[h]k[h()h,h(d r/11M

0k

r

tqtq

Example (cont.)

Minkowski Similarity Is a L-1 metric

where Ik and Jk is the number of pixels in bin k for image I and J

Distance between above three images D(H1, H2) = 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 = 8

D(H1, H3) = 6 + 6 + 2 + 2 + 2 + 2 + 2 + 2 = 24

D(H2, H3) = 5 + 5 + 3 + 3 + 1 + 1 + 1 + 1 = 23

n

kkkJI JIHHD

1

||),(

Example (cont.)

Minkowski Similarity Is a L-2 metric Distance between above three images

D(H1, H2) = (1 + 1 + 1 + 1 + 1 + 1 + 1 + 1)1/2 = 2.8

D(H1, H3) = (36 + 36 + 4 + 4 + 4 + 4 + 4 + 4)1/2 = 9.8

D(H2, H3) = (25 + 25 + 9 + 9 + 1 + 1 + 1 + 1) 1/2 = 8.5

QBIC distance Weighted Euclidean distance (QBIC)

Is a L-2 metric(?)distance between histogram H1 and H2:

D = (H1 - H2)T A (H1 - H2)

where A is a symmetric color similarity matrix

A (i, j) = 1 – d (ci, cj) / dmax

where ci and cj are the i-th and j-th color bins,

d (ci , cj) is the color distance in the color space, and dmax is the maximum distance between any two colors in the color space

Limitation

Ignore similarity between colors Example

Two color binsBin-1 color range: 1 – 10Bin-2 color range: 11 – 20

•Three color pixels

–Pixel 1 is Color 10 Bin-1

–Pixel 2 is Color 11 Bin-2

–Pixel 3 is Color 20 Bin-2

–Pixel 2 is similar to Pixel 3 than Pixel 1 unreasonable !

Limitation (cont.)

Ignore spatial relationships among pixels

Different image with same histogram

32

Noise-Free Queries (NFQ’s)

NFQ is more precise.

User can specify semantic constraints:

Spatial constraints (relative distances) Scaling constraints (relative sizes)

Rectangular query

Noise-freequery

Similar Less relevant

33

Challenges

How do we extract features if we do not know the matching areas beforehand ?

How do we index the images ?

Noise-freequery

34

One Solution – Local Color Histogram (LCH)

Each subimage has a color histogram.

Any combination of the histograms can be selected for comparison with the corresponding color histograms of the query image.

Query image

Database image

35

Limitations of LCH

Dilemma:

Using large partitions is not precise

Using small partitions is too expensive

Limitation:

difficult to handle scaling

Query image

Database image

36

Resultados esperados de uma boa CBIR com segmentação

2 3 216 396

2 5 12 183Query

Query

4

37

DEMOS

Hermitage Museum Web Site (QBIC)http://hermitagemuseum.org/

http://hermitagemuseum.org/fcgi-bin/db2www/qbicColor.mac/qbic?selLang=English

http://www.aa-lab.cs.uu.nl/cbirsurvey/cbir-

survey/