content-based image retrievalweb.cecs.pdx.edu/~maier/cs510iri/lectures/mitchell-07.pdf · 3 •...
TRANSCRIPT
![Page 1: Content-Based Image Retrievalweb.cecs.pdx.edu/~maier/cs510iri/Lectures/mitchell-07.pdf · 3 • Typical distance measure: Inner product m i m ⋅ =FiQ +Fi Q +...+F Q F Q 1 1 2 2 Some](https://reader036.vdocument.in/reader036/viewer/2022081408/607163820c77294b211219be/html5/thumbnails/1.jpg)
1
Content-Based Image Retrieval
Melanie Mitchell
A sample of existing systems
• Google image search
• Google image labeler
Why not just use text?
•QBIC at the Hermitage Museum
•WebSEEk
•Simplicity
•WISE
•Riya
![Page 2: Content-Based Image Retrievalweb.cecs.pdx.edu/~maier/cs510iri/Lectures/mitchell-07.pdf · 3 • Typical distance measure: Inner product m i m ⋅ =FiQ +Fi Q +...+F Q F Q 1 1 2 2 Some](https://reader036.vdocument.in/reader036/viewer/2022081408/607163820c77294b211219be/html5/thumbnails/2.jpg)
2
Possible applications
Query Image
Extract Features(Primitives)
Image Database
Features Database
SimilarityMeasure
MatchedResults
RelevanceFeedbackAlgorithm
From http://www.amrita.edu/cde/downloads/ACBIR.ppt
Basic technique
Each image in database is represented by a feature vector: (F1, F2, ...Fn)
Query is represented in terms of same features: Q
Goal: Find stored image with vector Fi most similar to query vector Q
![Page 3: Content-Based Image Retrievalweb.cecs.pdx.edu/~maier/cs510iri/Lectures/mitchell-07.pdf · 3 • Typical distance measure: Inner product m i m ⋅ =FiQ +Fi Q +...+F Q F Q 1 1 2 2 Some](https://reader036.vdocument.in/reader036/viewer/2022081408/607163820c77294b211219be/html5/thumbnails/3.jpg)
3
• Typical distance measure: Inner product
mi
miii QFQFQF +++=⋅ ...2211QF
Some issues
• Query format, easy of querying
• Speed
• Crawling, preprocessing
• Interactivity, user relevance feedback
• Visual features — which to use? How to combine?
• Curse of dimensionality
• Indexing
• Evaluation of performance
![Page 4: Content-Based Image Retrievalweb.cecs.pdx.edu/~maier/cs510iri/Lectures/mitchell-07.pdf · 3 • Typical distance measure: Inner product m i m ⋅ =FiQ +Fi Q +...+F Q F Q 1 1 2 2 Some](https://reader036.vdocument.in/reader036/viewer/2022081408/607163820c77294b211219be/html5/thumbnails/4.jpg)
4
Types of Features
• Text
• Color
• Texture
• Shape
• Layout
Color representations
• Histograms (global and regional) , correlograms
From: http://www.cse.ucsc.edu/classes/ee264/Winter02/xgfeng.ppt
QueryQuery
TargetTarget
Correlogram method: 1st
Histogram method: 31st
![Page 5: Content-Based Image Retrievalweb.cecs.pdx.edu/~maier/cs510iri/Lectures/mitchell-07.pdf · 3 • Typical distance measure: Inner product m i m ⋅ =FiQ +Fi Q +...+F Q F Q 1 1 2 2 Some](https://reader036.vdocument.in/reader036/viewer/2022081408/607163820c77294b211219be/html5/thumbnails/5.jpg)
5
• Moments of color distribution (mean, variance, skewness)
From: http://www2.cmp.uea.ac.uk/Research/compvis/ImageRetrieval/ImageRetrieval.htm
RGB distribution
Texture representations
• entropy
• contrast
• Fourier and wavelet transforms
• Gabor filters
![Page 6: Content-Based Image Retrievalweb.cecs.pdx.edu/~maier/cs510iri/Lectures/mitchell-07.pdf · 3 • Typical distance measure: Inner product m i m ⋅ =FiQ +Fi Q +...+F Q F Q 1 1 2 2 Some](https://reader036.vdocument.in/reader036/viewer/2022081408/607163820c77294b211219be/html5/thumbnails/6.jpg)
6
Feature Extraction [7] -- Texture
Grey – level co-occurrence matrix
http://www.amrita.edu/cde/downloads/ACBIR.ppt
From: http://www.mathworks.com/matlabcentral/files/9554/content/wavelets/tp2.html
![Page 7: Content-Based Image Retrievalweb.cecs.pdx.edu/~maier/cs510iri/Lectures/mitchell-07.pdf · 3 • Typical distance measure: Inner product m i m ⋅ =FiQ +Fi Q +...+F Q F Q 1 1 2 2 Some](https://reader036.vdocument.in/reader036/viewer/2022081408/607163820c77294b211219be/html5/thumbnails/7.jpg)
7
Shape representations
Need segmentation (another whole story!)
• area, eccentricity, major axis orientation
• Skeletons, shock graphs
• Fourier transformation of boundary
• histograms of edge orientations
From: http://www.lems.brown.edu/vision/researchAreas/ShockMatching/shock-ed-match-results1.gif
![Page 8: Content-Based Image Retrievalweb.cecs.pdx.edu/~maier/cs510iri/Lectures/mitchell-07.pdf · 3 • Typical distance measure: Inner product m i m ⋅ =FiQ +Fi Q +...+F Q F Q 1 1 2 2 Some](https://reader036.vdocument.in/reader036/viewer/2022081408/607163820c77294b211219be/html5/thumbnails/8.jpg)
8
From: http://www.cs.ucl.ac.uk/staff/k.jacobs/teaching/prmv/Edge_histogramming.jpg
Issues in feature-vector matching
Dimension reduction (e.g., by Principle Components Analysis)
5 Principal Components
![Page 9: Content-Based Image Retrievalweb.cecs.pdx.edu/~maier/cs510iri/Lectures/mitchell-07.pdf · 3 • Typical distance measure: Inner product m i m ⋅ =FiQ +Fi Q +...+F Q F Q 1 1 2 2 Some](https://reader036.vdocument.in/reader036/viewer/2022081408/607163820c77294b211219be/html5/thumbnails/9.jpg)
9
Feature selection (e.g., by information gain of feature)
Two different images might not have same number of features!
• cf. Pyramid matching (Grauman & Darrell, 2005)
![Page 10: Content-Based Image Retrievalweb.cecs.pdx.edu/~maier/cs510iri/Lectures/mitchell-07.pdf · 3 • Typical distance measure: Inner product m i m ⋅ =FiQ +Fi Q +...+F Q F Q 1 1 2 2 Some](https://reader036.vdocument.in/reader036/viewer/2022081408/607163820c77294b211219be/html5/thumbnails/10.jpg)
10
Sets of features
From: http://people.csail.mit.edu/kgrauman/slides/pyr_match_iccv2005.ppt
Pyramid match
optimal partial matching
From: http://people.csail.mit.edu/kgrauman/slides/pyr_match_iccv2005.ppt
![Page 11: Content-Based Image Retrievalweb.cecs.pdx.edu/~maier/cs510iri/Lectures/mitchell-07.pdf · 3 • Typical distance measure: Inner product m i m ⋅ =FiQ +Fi Q +...+F Q F Q 1 1 2 2 Some](https://reader036.vdocument.in/reader036/viewer/2022081408/607163820c77294b211219be/html5/thumbnails/11.jpg)
11
Multidimensional Indexing
• Compare with text indexing. Much higher dimensionality. Complex similarity measures
• Examples of indexing techniques:– k-d tree– r*-tree
From: http://www.cse.ogi.edu/class/cse580ir/handouts/14%20November/Multimedia%20Information%20Retrieval/img016.gif
![Page 12: Content-Based Image Retrievalweb.cecs.pdx.edu/~maier/cs510iri/Lectures/mitchell-07.pdf · 3 • Typical distance measure: Inner product m i m ⋅ =FiQ +Fi Q +...+F Q F Q 1 1 2 2 Some](https://reader036.vdocument.in/reader036/viewer/2022081408/607163820c77294b211219be/html5/thumbnails/12.jpg)
12
Visual abilities largely missing from current CBIR systems
• object recognition
• perceptual organization
• similarity between semantic concepts
“The semantic gap”
“In general, current systems have not yet had significant impact on society due to an inability to bridge the semantic gap between computers and humans.”
![Page 13: Content-Based Image Retrievalweb.cecs.pdx.edu/~maier/cs510iri/Lectures/mitchell-07.pdf · 3 • Typical distance measure: Inner product m i m ⋅ =FiQ +Fi Q +...+F Q F Q 1 1 2 2 Some](https://reader036.vdocument.in/reader036/viewer/2022081408/607163820c77294b211219be/html5/thumbnails/13.jpg)
13
How to bridge the semantic gap?