indexing and retrieval
DESCRIPTION
Indexing and Retrieval. James Hill, Ozcan Ilikhan, Mark Lenz {jshill4, ilikhan, mlenz} @cs.wisc.edu. 1. Presentation Outline. 1- Introduction 2- Common methods used in the papers * SIFT descriptor * k-means clustering * TF-IDF weight 3- Video Google - PowerPoint PPT PresentationTRANSCRIPT
CS 766: Computer VisionComputer Sciences Department, University of Wisconsin-Madison
Indexing and RetrievalJames Hill, Ozcan Ilikhan, Mark Lenz
{jshill4, ilikhan, mlenz} @cs.wisc.edu
1
Presentation Outline
1- Introduction
2- Common methods used in the papers* SIFT descriptor* k-means clustering* TF-IDF weight
3- Video Google
4- Scalable Recognition with a Vocabulary Tree
5- City-Scale Location Recognition
2
Introduction
Find identical objects in multiple images
Difficulties with changes in– Scale– Orientation– Viewpoint– Lighting
Search time and storage space
3
Indexing and Retrieval
Common Solutions
Invariant features (e.g. SIFT)
kd-trees
Best Bin First
4
SIFT - Scale-Invariant Feature Transform
Key Steps
5
1)Difference of Gaussians in scale space
2)Maxima and minima are feature points
3)Remove low-contrast and non-robust edge points
4)Assign each point an orientation
5)Create a descriptor from windowed region
SIFT - Scale-Invariant Feature Transform
Key Benefits
6
Feature points invariant to scale and translation Orientations provide invariance to rotation Distinctive descriptors are partially invariant to changes
in illumination and viewpoint Robust to background clutter and occlusion
k-means clustering
Motivation (what are we trying to do)
We want to develop a method for finding the centers of different clusters in a set of data.
7
k-means clustering
8
k-means clustering
9
k-means clustering
10
k-means clustering
11
k-means clustering
How do we find these means?
We need to perform a minimization on:
12
k
i Sxij
ij
x1
2
k-means clustering
How do we extend this?
With Hierarchical k-means Clustering!
13
k-means clustering
14
k-means clustering
15
k-means clustering
16
k-means clustering
Now that we can cluster our data, how can we use this information to quickly find the closest vector in our data given some test vector?
17
k-means clustering
We will build a vocabulary tree using this clustering method.
Each vector in our data (including the means) will be considered a “word” in our vocabulary.
We will build a tree using the means of our data.
18
k-means clustering
19
k-means clustering
20
k-means clustering
21
TF-IDF
Term frequency–inverse document frequency (tf–idf): is a statistical measure used to evaluate how important a word is to a document in a collection or corpus.
A standard weight often used in information retrieval and text mining.
22
TF-IDF
23
nid : the number of occurrences of word i in document d.
nd : the total number of words in document d.
Ni : the number of documents containing term i.
N : the total number of documents in the whole database.
TF-IDF
24
word frequency inverse document frequency
X
Each document is represented by a vector
Then vectors are organized as an inverted file.
TF-IDF
25
Image credit: http://www.lovdata.no/litt/hand/hand-1991-2.html
Video Google
26
A Text Retrieval Approach to Object Matching in Videos
Josef Sivic and Andrew ZissermanVisual Geometry Group,
Department of Engineering ScienceUniversity of Oxford, United Kingdom
Proceedings of the International Conference on Computer Vision (2003)
Video Google
27
Efficient Visual Search of Videos Cast as Text Retrieval
IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 31, Number 4, page 591--606, 2009
Fundamental idea of paper:
Retrieve key frames and shots of a video containing a particular object with ease, speed, and accuracy with which Google retrieves text documents (web pages) containing particular words.
Video Google
28
Recall Text Retrieval (preprocessing)
1. Parse documents into words
2. Stemming: “walk" = { “walk,” “walking,” “walks”,…}
3. Stop list to reject very common words , such as “the” and “an”.
4. Each document is represented by a vector with components given by
the frequency of occurrence of the words the document contains
5. Store vector in an inverted file.
Video Google
29
Can we treat video the same way?
What and where are the words of a video?
Video Google
30
1. Detect affine covariant regions in each key frame of video
2. Reject unstable regions.
3. Build visual vocabulary
4. Remove stop listed words
5. Compute weighted document frequency
6. Build the index (inverted file).
The Video Google algorithm:a) Pre-processing (off-line):
Video Google
31
Building a Visual Vocabulary
Step 1. Calculate viewpoint invariant regions:
Shape Adapted (SA) region: centered on corner-like features
Maximally Stable (MS) region: correspond to blobs of high
contrast with respect to their surroundings such as a dark window
on a gray wall.
Each region is represented by a 128-dimentional vector using SIFT descriptor
720 x 576 pixel video frame ≈ 1200 regions
Video Google
32
Video Google
33
Step 2. Reject unstable regions:
Any region that does not survive for more than 3 frames is rejected.
This “stability check” significantly reduces the number of regions to
about 600 per frame.
Video Google
34
Step 3. Build Visual Vocabulary:
Use K-Means clustering to vector quantize descriptors into clusters
Mahalanobis distance:
Video Google
35
Step 4. Remove stop-listed visual words:
The most frequent visual words that occur in almost all images,
such as highlights which occur in many frames, are rejected.
Video Google
36
Step 5. Compute tf-idf weighted document frequency vector:
Variations of tf-idf may be used.
Step 6. Build inverted-file indexing structure:
Video Google
37
1. Determine the set of visual words within the query region
2. Retrieve keyframes based on visual word frequencies
3. Re-rank the top keyframes using spatial consistency
The Video Google algorithm:b) Run-time (on-line):
Video Google
38
Matched covariant regions in the retrieved frames should have a similar
spatial arrangement to those of the outlined region in the query image.
Spatial consistency:
Video Google
39
How it works:
Query region and its close-up.
Video Google
40
How it works:
Original matches based on visual words
Video Google
41
How it works:
Original matches based on visual words
Video Google
42
How it works:
Matches after using the stop-list
Video Google
43
How it works:
Final set of matches after filtering on spatial consistency
Video Google
44
Video Google
45
Video Google
46
Real-time demo
CS 766: Computer VisionComputer Sciences Department, University of Wisconsin-Madison
Scalable Recognition With a Vocabulary
TreeJames Hill, Ozcan Ilikhan, Mark Lenz
{jshill4, ilikhan, mlenz} @cs.wisc.edu
47
The Paper
Scalable Recognition with a vocabulary treeDavid Nister and Henrik Stewenius
Center for Visualization and Virtual Environments
Department of Computer Science, University of Kentucky
Published in 2006
Appeared in: 2006 IEEE Computer Science Conference on Computer Vision and Pattern Recognition
48
What are we trying to do.
Provide an indexing scheme that:
Scales to large image databases (1 million).
Retrieves images in an acceptable amount of time.
49
Inspiration
Sivic and Zisserman (what you just saw)
Used k-means to partition the descriptors in several pictures.
Used TF-IDF to score an image and find a close match.
50
What’s new?
The idea of a vocabulary tree.
Using a larger vocabulary tree speeds things up and improve match quality
Can use many more training images (35000 vs 400)
Can insert new images into the Database quickly (0.2s vs 10s)
51
How do we do it?
Follow these three steps:
1. Build the vocabulary tree using the image descriptors.
2. Generate a score for a given query image.
3. Find the images in the database that best match that score.
52
Recap the Vocabulary Tree
1. For each image in our database, we calculate a set of feature point descriptors.
2. Each of these descriptors is a vector of numbers which exists in some space (128).
3. Consider each of these vectors to be a “word” in the vocabulary of our database.
53
Recap the Vocabulary Tree
Build the vocabulary tree using hierarchical k-means clustering.
54
Recap the Vocabulary Tree
55
Recap the Vocabulary Tree
56
Recap the Vocabulary Tree
57
What’s it good for?
Now that we have a vocabulary tree, we can generate a path down the vocabulary tree which is stored in an integer for scoring.
At each level of the tree, the descriptor is compared to each of the k children using a dot product. The closest is the path that is followed.
58
Scoring
We have a bunch of paths through the tree, how do we compare the query image to a database image?
At each node, we define a weight wi.
The paper suggests two methods
• Use a constant weighting scheme.
• Use an entropy weighting scheme such as
59
iN
Nln
Scoring (continued)
Where
N is the number of images in the database
Ni is the number of images in the database with at least one descriptor vector path through node i.
60
iN
Nln
Scoring (continued)
This scoring mechanism results in a TF-IDF scheme.
So we should see a higher score if more nodes are shared by more descriptors.
61
Scoring (continued)
To compare two scores, we use the normalized difference between the query score and the database score.
62
d
d
q
qdqs ),(
Scoring (continued)
Researchers found that the most important factors to quality where.
• A large vocabulary tree.
• Stronger weights towards the leaves of the tree.
• Using the L1 norm in the previous equation.
63
Scoring Implementation
Scoring is implemented using inverted files
• At each node create an inverted file
• Each file contains a list of images in which the current node appears.
• The inverted file of inner nodes is simply the concatenation of it’s children’s inverted files.
• Database image scores are pre-computed and pre normalized.
64
Testing
This method was tested using a a database of 40000 CD album covers.
Pictures of cd album covers where then used as query images and run against the database.
Also tested using 6376 images in groups of 4.
Each image was queried in the hopes that the other 3 images would produce the top scores.
Have tested on databases with image counts as high as 1 million (highest at time of writing)
65
Testing
66
Results
67
Conclusions
The main conclusions of the paper are:
• Using a larger vocabulary tree makes things better.
• Using an L1 norm in the normalized difference of the scores produces better results than the L2 norm
• This method can scale up to 1 million images and still run in near real time.
68
CS 766: Computer VisionComputer Sciences Department, University of Wisconsin-Madison
City-Scale Location Recognition
James Hill, Ozcan Ilikhan, Mark Lenz{jshill4, ilikhan, mlenz} @cs.wisc.edu
69
City-Scale Location Recognition
70
Estimate location by matching features from a large set of images
City-Scale Location Recognition
71
City-wide database of photos labeled with location
Image Features
72
SIFT features invariant to– Translation– Scale– Orientation– Illumination (partially)
Difficulties Matching Features
73
Storage space– 30,000 images ≈ 100,000,000 SIFT features ≈ 12 GB
Search time
kd-trees and Best Bin First require descriptors
Method
74
Cluster features into visual words
Build vocabulary tree from clusters
Search tree to score matches
Location of image with top score
Method
75
Build trees with informative features
Create trees of varying branching factor
Vary number of comparisons during search
Vocabulary Tree
76
Visual word = region of an object
Just need the distance between a query feature and each node
Only leaf nodes are words
Informative Features
77
Cluster small subsets into visual words
Compute information gain of features
Select most informative features to build tree
Information Gain
78
Informative Feature– Found in all images of a location– Not in any image of another location
Information gain: measure of how much new information reduces uncertainty
Information Gain
79
N DB=number of images in databaseN L=number of images at location lia=number of images visual wordw j occurs at location lib=number of images visual wordw j occurs at other locations
Building the Tree
80
Hierarchical k-means to cluster features
Nodes are the centroids
Leaves are the visual words
Branching Factor
81
Vary number of nodes compared to increase search accuracy
Fixed vocabulary size M
Branching factor k, depth L
kL≈M
Greedy N-Best Paths
82
Approximate nearest neighbor
Similar to Best Bin First
Generalization of vocab tree search
Search multiple branches at each level
Greedy N-Best Paths
83
k + kN(L-1) comparisons
Matching
84
Votes for image d = Cd
Computed in linear time in # of features
Results
85
30,000 images covering 20 km
278 GPS-labelled query images
Performance = % query images within 10m of ground truth
Results
86
Informative Features vs. Uniform
Results
87
Greedy N-Best Paths vs. Best Bin First
Results
88
Top n matches
Conclusion
89
Vocabulary tree structure affects performance in recognition tasks
Structure becomes more critical as database size increases
Number of comparisons drives performance, not branching factor
CS 766: Computer VisionComputer Sciences Department, University of Wisconsin-Madison
Q & A, DiscussionMonday, November 29, 2010
CS 766: Computer VisionComputer Sciences Department, University of Wisconsin-Madison
AcknowledgementsMany thanks to Prof. Andrew Zisserman and
Dr. Josef Sivic for providing us with extra materials for presentation.
91