![Page 1: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/1.jpg)
ICIP 2004, Singapore, October 25-27
A
Comparison of
Continuous vs. Discrete Image Modelsfor
Probabilistic Image and Video Retrieval
Arjen P. de Vries and Thijs Westerveld
![Page 2: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/2.jpg)
ICIP 2004, Singapore, October 25-27
Theory
![Page 3: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/3.jpg)
ICIP 2004, Singapore, October 25-27
Generative Models…
• A statistical model for generating data– Probability distribution over samples in a
given ‘language’M
P ( | M ) = P ( | M )
P ( | M, )
P ( | M, )
P ( | M, )
© Victor Lavrenko, Aug. 2002
aka
‘Language Modelling’
![Page 4: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/4.jpg)
ICIP 2004, Singapore, October 25-27
• Basic question:– What is the likelihood that this document is
relevant to this query?
• P(rel|I,Q) = P(I,Q|rel)P(rel) / P(I,Q)
… in Information Retrieval
• P(I,Q|rel) = P(Q|I,rel)P(I|rel)
![Page 5: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/5.jpg)
ICIP 2004, Singapore, October 25-27
Retrieval (Query generation)Models
P(Q|M1)
P(Q|M4)
P(Q|M3)
P(Q|M2)
Query
Docs
![Page 6: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/6.jpg)
ICIP 2004, Singapore, October 25-27
‘Language Modeling’
• Not just ‘English’• But also, the
language of– author– newspaper– text document– image
• Shakespeare or Dickens?
• Indeed the short and the long. Marry, ‘tis a noble Lepidus.
![Page 7: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/7.jpg)
ICIP 2004, Singapore, October 25-27
‘Language Modeling’
• Guardian or Times?• Not just ‘English’• But also, the
language of– author– newspaper– text document– image
![Page 8: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/8.jpg)
ICIP 2004, Singapore, October 25-27
‘Language Modeling’
• or ?
• Not just English!• But also, the
language of– author– newspaper– text document– image
![Page 9: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/9.jpg)
ICIP 2004, Singapore, October 25-27
The Fundamental Problem• Usually, we don’t know the model M
– But have a sample representative of that model
• First estimate a model from a sample
• Then compute the observation probability
P ( | M ( ) )
M© Victor Lavrenko, Aug. 2002
![Page 10: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/10.jpg)
ICIP 2004, Singapore, October 25-27
• Urn metaphor
Unigram Language Models
© Victor Lavrenko, Aug. 2002
• P( | ) ~ P ( | ) P ( | ) P ( | ) P ( | )
= 4/9 * 2/9 * 4/9 * 3/9
![Page 11: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/11.jpg)
ICIP 2004, Singapore, October 25-27
The Zero-frequency Problem
• Suppose some event not in our example– Model may assign zero probability to that
event– And to any set of events involving the
unseen event
?
![Page 12: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/12.jpg)
ICIP 2004, Singapore, October 25-27
Smoothing
• Idea: shift part of probability mass to unseen events
• Interpolation with background model– Reflects expected frequency of events– Plays role of IDF (inverse document freq.)
+(1-)
![Page 13: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/13.jpg)
ICIP 2004, Singapore, October 25-27
The IDF Role of Smoothing
P(x| ) +(1-) P(x| )
P(x| )• = +1
(1-) P(x| )
– Ranking independent of
![Page 14: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/14.jpg)
ICIP 2004, Singapore, October 25-27
Practise
![Page 15: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/15.jpg)
ICIP 2004, Singapore, October 25-27
• Pixel level: no semantics
• Pixel blocks/regions
Image Retrieval
![Page 16: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/16.jpg)
ICIP 2004, Singapore, October 25-27
Modelling Images
• Compute local features– Eg., blueness and yellowness
0.2567 0.3294
0.1334 0.1664 0.3125 0.3714 0.3288 0.4624 0.1854 0.2308
. .
. .
. .
![Page 17: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/17.jpg)
ICIP 2004, Singapore, October 25-27
![Page 18: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/18.jpg)
ICIP 2004, Singapore, October 25-27
Discrete Model
yellow
blue
![Page 19: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/19.jpg)
ICIP 2004, Singapore, October 25-27
Discrete Model
![Page 20: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/20.jpg)
ICIP 2004, Singapore, October 25-27
Modelling Images
blue
yellow
Histogram also models empty regions in the feature space
Boundaries are hard
![Page 21: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/21.jpg)
ICIP 2004, Singapore, October 25-27
Continuous Model
• Build Gaussian Mixture model using expectation maximisation (EM)
• 2 Components– Centers, covariance– Random intialisation blue
yellow
![Page 22: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/22.jpg)
ICIP 2004, Singapore, October 25-27
Continuous Model
![Page 23: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/23.jpg)
ICIP 2004, Singapore, October 25-27
Discrete vs. Continuous
• Discrete Model– Low indexing cost (binning)– Low retrieval cost (inverted file)– But… how to partition the indexing space?
• Continuous Model– High indexing cost (EM algorithm)– High retrieval cost (access all data)– But… less overfitting better generalisation
![Page 24: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/24.jpg)
ICIP 2004, Singapore, October 25-27
Experiments
• TRECVID2003 search task– Discrete vs. Continuous– Regions vs. full Query examples– All examples vs. designated only
• Mean average precision
![Page 25: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/25.jpg)
ICIP 2004, Singapore, October 25-27
Results
• Continuous Model significantly better on almost all queries
• However, Discrete Model significantly better for small number of highly focused queries (e.g., flames, airplane taking off)– More analysis needed though
![Page 26: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/26.jpg)
ICIP 2004, Singapore, October 25-27
Conclusions
• Language modelling approach to IR also applicable to retrieval of other media
• Discrete vs. Continuous Model– Continuous Model almost always better– Unfortunately, Discrete Model far easier to
implement efficiently
![Page 27: ICIP 2004, Singapore, October 25-27 A Comparison of Continuous vs. Discrete Image Models for Probabilistic Image and Video Retrieval Arjen P. de Vries](https://reader035.vdocument.in/reader035/viewer/2022062518/56649f575503460f94c7cc6b/html5/thumbnails/27.jpg)
ICIP 2004, Singapore, October 25-27
Future Work
• Improve Sampling Process– Better texture representation?– Overlapping, multi-scale image patches?
• Improve Discrete Model– Partitioning of feature space in grid cells
• Compare the performance of the two models in interactive setting with relevance feedback– Higher quality per iteration vs. many iterations