similar image search
DESCRIPTION
similar image searchTRANSCRIPT
![Page 1: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/1.jpg)
Similar Image Search with a Tiny Bag-of-Delegates
Representation
Weiwen Tu, Rong Pan, Jingdong Wang
![Page 2: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/2.jpg)
Similar Image Search
• text-based image search• context-based image search
![Page 3: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/3.jpg)
Tiny Bag-of-Delegates
Given a large image set ,
our goal is to find a bag-of-delegates to represent each image.
is represented by M delegates,
![Page 4: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/4.jpg)
Tiny Bag-of-Delegates
So we need to build M vocabularies ,
where with J being the size of each vocabulary.
From each vocabulary an element is select as a delegate for each image.
![Page 5: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/5.jpg)
Tiny Bag-of-Delegates
To conduct search, we organize the image set using M inverted indices ,each corresponding to a vocabulary respectively.
An inverted index is composed of J list,
each list is corresponding to a word, consists of images that are mapped to the word ( corresponds to )
![Page 6: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/6.jpg)
Example for Tiny Bag-of-Delegates
𝐼 1→(𝑣1 ,𝑣3 ,𝑣5)
𝐼 2→(𝑣1 ,𝑣4 ,𝑣6)
𝐼 3→(𝑣2 ,𝑣3 ,𝑣6)
𝑉 3𝑉 2𝑉 1
𝑇 1
𝑇 2
𝑇 3
Query:
![Page 7: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/7.jpg)
Tiny Bag-of-Delegates
Given an image whose bag-of-delegates is defined as , we find the lists
where is a function mapping from the delegate w to the list in the inverted index . Then regard all the images in these lists as the candidate similar images, and finally order these images according to their similarities computed with their features.
![Page 8: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/8.jpg)
Spatial Partition Tree
• Spatial partition tree is a method for recursively subdividing a space into subsets by hyperplanes.
• Popular data structures: Kd-tree, PCA-tree, VP-tree.
![Page 9: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/9.jpg)
Example for Spatial Partition Tree
The left figure shows a spatial partitioning induced by an RP tree. The cross is q query point and the lines means the partition hyperplanes.
𝒗𝟏
𝒗𝟐
𝒗𝟑𝒗𝟒
𝒗𝟓
𝒗𝟔𝒗𝟕
𝒗𝟖
![Page 10: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/10.jpg)
Tree Construction
procedure MAKETREE(S)if(S) < MinSize
then return(Leaf)else;
return ([Rule, LeftTree, RightTree])
We define the split method as ChooseRule. The core tree-building is called MakeTree, and takes as input a data set .
![Page 11: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/11.jpg)
Problem
A larger amount of trees yield a better performance, while requiring more storage to save the inverted indices and accordingly more query time
Problem:Can we use a small amount of trees but with the search performance guaranteed?
![Page 12: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/12.jpg)
Vocabulary Construction
Our idea is to use supervised information in constructing trees. In similar image search, the supervised information is the true nearest neighbors of each image.
We denote the true neighbors of an image by aList ,
where is the index of a similar image.
![Page 13: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/13.jpg)
Criterion
The nearest neighbor candidates discovered from the bag-of-delegates representation can be written as
The recall of :The average recall :
![Page 14: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/14.jpg)
Criterion
Given a set of Z candidate inverted indices
the goal is to find inverted indices such that the recall is maximized. The objective function is written as
where .
![Page 15: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/15.jpg)
Forward selection scheme
We progressively generate the candidate inverted indices for each step. At the beginning, we randomly generate a set of candidate inverted indices
,We evaluate each inverted index to compute the recall, and then identify the first inverted index that corresponds to the largest recall. Denote the identified inverted index as and the current solution is
![Page 16: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/16.jpg)
Forward selection scheme
The later steps sequentially find the inverted indices oneby one. Considering the step, we have found t indices .We generate a set of new candidates
and form the whole candidates
The objective of identifying the (t+ 1)-th index is as follows,
![Page 17: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/17.jpg)
Random selection vs. forward selection
Random selection
Forward selection
![Page 18: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/18.jpg)
Adaptive scheme
We maintain a set
where and are truly neighboring points but not appear in the same bucket in the previously-identified indices .
Compute top r sparse principal directions from sample points,
![Page 19: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/19.jpg)
Adaptive scheme
Each direction can form a partition ,
where v is the current node with data points
Then we judge a pair is in the same side
![Page 20: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/20.jpg)
Adaptive scheme
Finally, the objective function of the direction is,
![Page 21: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/21.jpg)
Adaptive forward selection
1. Initialization: 𝑄←𝐿,𝑡 ←0,𝑒←ȁ𝑄ȁ,𝑅←∅
2. Repeat
3. Candidate proposal
Randomly generate spatial partition tree 𝒯ത per Q
4. Candidate selection
Choose the spatial partition tree T from 𝒯ത that keeps the large number of Q
5. Update
Discard all the pairs of points lying in the same bucket in T from Q 𝑡 ←𝑡+ 1,𝑒←ȁ𝑄ȁ,𝑅←𝑅∪{𝑇}
6. Until 𝑒≤ 𝜀 && 𝑡 ≥ 𝜏
7. Return R;
![Page 22: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/22.jpg)
EXPERIMENTS
• Data set– 32 ×32 color images– 1M tiny images to form around 80M
images– global GIST descriptor(384 D vector)
• Evaluation criteria(average accuracy score)
![Page 23: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/23.jpg)
Evaluation
Given an image , the accuracy is computed as,
: true neighbors list: the top K images found from M inverted indices, The whole average accuracy is computed as
![Page 24: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/24.jpg)
Accuracy vs indices
Bucket size: the maximum number of points in a leaf node;g: the number of target NNs.
![Page 25: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/25.jpg)
Accuracy vs indices
bucket size = 300
bucket size = 100
![Page 26: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/26.jpg)
accuracy vs accessed images
![Page 27: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/27.jpg)
Out-of-sample Test
![Page 28: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/28.jpg)
Visual search result
13 : 9 22 : 9
![Page 29: Similar image search](https://reader035.vdocument.in/reader035/viewer/2022062703/555095c3b4c9058b208b45ca/html5/thumbnails/29.jpg)
Thank you!