content-based image retrieval qbic homepage the state hermitage museum ...

Content-Based Image Content-Based Image RetrievalRetrieval

• QBIC Homepagehttp://wwwqbic.almaden.ibm.com/

• The State Hermitage Museumhttp://www.hermitagemuseum.org/fcgi-bin/ db2www/qbicSearch.mac/qbic?selLang=English

Query By Keyword: Some textual attributes (keywords) should be maintained for each image. The image can be indexed according to these attributes, so that they can be rapidly retrieved when a query is issued. This type of query can be expressed in Structured Query Language (SQL).

Query By Example (QBE): User just show the system a sample image, then the system should be able to return similar images or images containing similar objects.

Image Database QueriesImage Database Queries

Image Distance & Image Distance & Similarity MeasuresSimilarity Measures

1. Color Similarity

2. Texture Similarity

3. Shape Similarity

4. Object & Relationship similarity

Color SimilarityColor Similarity

• Color percentages matching: R:20%, G:50%, B:30%

• Color histogram matching

Dhist(I,Q)=(h(I)-h(Q))TA(h(I)-h(Q))

A is a similarity matrix colors that are very similar should have similarity values close to one.

Color Color ExampleExample

• Color layout matching: compares each grid square of the query to the corresponding grid square of a potential matching image and combines the results into a single image distance

where CI(g) represents the color in grid square g of a database image I and CQ(g) represents the color in the corresponding grid square g of the query image Q. some suitable representations of color are

1. Mean2. Mean and standard deviation3. Multi-bin histogram

Color Layout SimilarityColor Layout Similarity

g

QIcolorcolorgridded gCgCdQId ))(),((ˆ),(_

Color Color LayoutLayoutExampleExample11

Color Layout Example2Color Layout Example2

• Pick and clickSuppose T(I) is a texture description vector which is a vector of numbers that summarizes the texture in a given image I (for example: Laws texture energy measures), then the texture distance measure is defined by

• Texture layout

Texture SimilarityTexture Similarity

g

QItexturetexturegridded gTgTdQId ))(),((ˆ),(_

2

__ )()(min),( QTiTQId Iiclickandpick

IQ IQ basebased on d on Pick Pick and and ClickClick

Shape SimilarityShape Similarity

1. Shape Histogram

2. Boundary Matching

3. Sketch Matching

1. 1. Shape HistogramShape Histogram• Projection matchingProjection matching

• Horizontal & vertical projection: Each row and each column become a bin in the histogram. The count that is stored in a bin is the number of 1-pixels that appear in that row or column.• Diagonal projection: An alternative is to define the bins from the top left to the bottom right of the shape.

• Size invariant the number of row bins and the number of column bins in the bounding box can be fixed, histograms can be normalized before matching.• Translation invariant • Rotation invariant compute the axis of the best-fitting ellipse and rotate the shape

水平垂直投影水平垂直投影Horizontal and vertical Horizontal and vertical

projectionsprojections

H(i)

V(j)

對角投影 對角投影 (Diagonal (Diagonal projection)projection)

• Orientation histogramOrientation histogram

• Construct a histogram over the tangent angle at each pixel on the boundary of the shape.

• Size invariant histograms can be normalized before matching.• Translation invariant• Rotation invariant choosing the bin with the largest count to be the first bin.• Starting point invariant

1. 1. Shape HistogramShape Histogram

θ

# of pixels

2. 2. Boundary MatchingBoundary Matching

11D Fourier TransformD Fourier Transform on the boundary

1

0

2

1

0

2

10,

10,

N

n

N

knj

kn

N

k

N

knj

n

k

nnn

Nneau

NkN

eua

jyxu

• If only the first M coefficients (a0, a1, …, aM-1) are used, then

is an approximation of un

• the coefficients (a0, a1, …, aM-1) is called Fourier DescriptorsFourier Descriptors

• The Fourier distance measure is defined as:

1

0

2

10,ˆM

n

N

knj

kn Nneau

1

0

2),(

M

n

Qn

InFourier aaQId

Fourier DescriptorsFourier Descriptors

傅立葉描述元之特性傅立葉描述元之特性Properties of Fourier Properties of Fourier

DescriptorsDescriptors

Simple geometric transformations of a boundary, such as translation, rotation, and scaling, are related to simple operations of the boundary’s Fourier descriptors.

Transformation Boundary Fourier descriptor

Identity

Translation

Scaling or zooming

Starting point

Rotation

Reflection

)(nu

0)()(~ ununu

)()(~ nunu )()(~

0nnunu 0)()(~ jenunu

2)()(~ 2* jenunu

)(ka

)()()(~0 kukaka

)()(~ kaka

N

knj

eka02

)(~

0)()(~ jekaka

)(2)()(~ 2* kekaka j

11 constant M-k,a

aQk

Ik

A formula for Fourier Descriptor that is invariant to translation, scaling, rotation, and starting point.

A secret formulaA secret formula

IQ based IQ based on on Boundary Boundary MatchingMatching

3. 3. Sketch MatchingSketch Matching

For every image in the database to be compared, perform the following steps.

1. Affine transformation to specified size and applying median filter.

2. Edge detection using a gradient-based edge-finding algorithm Refined edge image

3. Thinning and shrinking Abstract image

4. The images are divided into grid squares and matching is performed based on local correlation.

The sketch distance measure is the inverse of the sum of each of the local correlations

where I(g) refres to grid square g of the abstract image I, Q(g) refers to grid square g of the linear sketch resulting from query image Q.

g nnsketch gQgIshiftnCorrelatio

QId)(,)(max

1),(

3. 3. Sketch MatchingSketch Matching

Object and Relational Object and Relational SimilaritySimilarity

• Ask for images containing certain objects, such as people or dogs.

• Face finding• Flesh finding

• Ask for images containing abstract concepts, such as happiness or beauty.

• Asks for objects with certain spatial relationships, such as cowboy riding a horse.

• construct a relational graph whose nodes represent objects and edges represent spatial relationships.

Detecting significant Detecting significant changes in videoschanges in videos

1. Scene change2. Shot change3. Camera pan4. Camera zoom5. Camera effects: fade, dissolve, and wipe

Segment and store video subsequences in digital libraries for random access.

Segmenting Video Segmenting Video SequencesSequences

• The transitions can be used to segment the video and can be detected by large changes in the features of the images over time.

MaxColMaxRow

crIcrIIId

MaxRow

r

MaxCol

c ttttpixel

1

0

1

0],[],[

),(

Similarity Similarity measure measure

by by histogramhistogram

Similarity measure by Similarity measure by likelihood ratiolikelihood ratio

• Break the image into larger blocks and test to see if a majority of the blocks are essentially the same in both images.

),(

otherwise 0

if 1),(

22 :ratio likelihood

21;

21

21

21

22

2121

2211

iiIBIBblock

rblock

BBd),Id(I

rBBd

vv

vv

r

ii