icip 2004, singapore, october 25-27 a comparison of continuous vs. discrete image models for...

ICIP 2004, Singapore, October 25-27

A

Comparison of

Continuous vs. Discrete Image Modelsfor

Probabilistic Image and Video Retrieval

Arjen P. de Vries and Thijs Westerveld


Theory


Generative Models…

• A statistical model for generating data– Probability distribution over samples in a

given ‘language’M

P ( | M ) = P ( | M )

P ( | M, )

P ( | M, )

P ( | M, )

© Victor Lavrenko, Aug. 2002

aka

‘Language Modelling’


Retrieval (Query generation)Models

P(Q|M1)

P(Q|M4)

P(Q|M3)

P(Q|M2)

Query

Docs


‘Language Modeling’

• Not just ‘English’• But also, the

language of– author– newspaper– text document– image

• Shakespeare or Dickens?

• Indeed the short and the long. Marry, ‘tis a noble Lepidus.



• Guardian or Times?• Not just ‘English’• But also, the




• or ?

• Not just English!• But also, the



The Fundamental Problem• Usually, we don’t know the model M

– But have a sample representative of that model

• First estimate a model from a sample

• Then compute the observation probability

P ( | M ( ) )

M© Victor Lavrenko, Aug. 2002


• Urn metaphor

Unigram Language Models

© Victor Lavrenko, Aug. 2002

• P( | ) ~ P ( | ) P ( | ) P ( | ) P ( | )

= 4/9 * 2/9 * 4/9 * 3/9


The Zero-frequency Problem

• Suppose some event not in our example– Model may assign zero probability to that

event– And to any set of events involving the

unseen event

?


Smoothing

• Idea: shift part of probability mass to unseen events

• Interpolation with background model– Reflects expected frequency of events– Plays role of IDF (inverse document freq.)

+(1-)


The IDF Role of Smoothing

P(x| ) +(1-) P(x| )

P(x| )• = +1

(1-) P(x| )

– Ranking independent of


Practise


• Pixel level: no semantics

• Pixel blocks/regions

Image Retrieval


Modelling Images

• Compute local features– Eg., blueness and yellowness

0.2567 0.3294

0.1334 0.1664 0.3125 0.3714 0.3288 0.4624 0.1854 0.2308

. .

. .

. .


Discrete Model

yellow

blue


Discrete Model


Modelling Images

blue

yellow

Histogram also models empty regions in the feature space

Boundaries are hard


Continuous Model

• Build Gaussian Mixture model using expectation maximisation (EM)

• 2 Components– Centers, covariance– Random intialisation blue

yellow


Continuous Model


Discrete vs. Continuous

• Discrete Model– Low indexing cost (binning)– Low retrieval cost (inverted file)– But… how to partition the indexing space?

• Continuous Model– High indexing cost (EM algorithm)– High retrieval cost (access all data)– But… less overfitting better generalisation


Experiments

• TRECVID2003 search task– Discrete vs. Continuous– Regions vs. full Query examples– All examples vs. designated only

• Mean average precision


Results

• Continuous Model significantly better on almost all queries

• However, Discrete Model significantly better for small number of highly focused queries (e.g., flames, airplane taking off)– More analysis needed though


Conclusions

• Language modelling approach to IR also applicable to retrieval of other media

• Discrete vs. Continuous Model– Continuous Model almost always better– Unfortunately, Discrete Model far easier to

implement efficiently


Future Work

• Improve Sampling Process– Better texture representation?– Overlapping, multi-scale image patches?

• Improve Discrete Model– Partitioning of feature space in grid cells

• Compare the performance of the two models in interactive setting with relevance feedback– Higher quality per iteration vs. many iterations

icip 2004, singapore, october 25-27 a comparison of continuous vs. discrete image models for...

Documents

model mbut

aka language modelling

indexing space

acomparison of continuous

idf role of smoothing

query examplesall examples

multiscale image patches

qrelprel pi