![Page 1: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/1.jpg)
A String Matching Approach for Visual Retrieval and ClassificationMei-Chen Yeh* and Kwang-Ting Cheng
Learning-Based Multimedia LabDepartment of Electrical and Computer EngineeringUniversity of California, Santa Barbara
![Page 2: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/2.jpg)
![Page 3: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/3.jpg)
Representation Based on Local Features
Belongie et al.; Berg. et al.; Ling & Jacobs Ozkan & Duyulu; Bicego et al.; Zou et al.
![Page 4: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/4.jpg)
Matching
• Features are unordered• Similarity is measured only on a feature subset
![Page 5: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/5.jpg)
Alternative Representation?
],...,,[ 21 mxxx
Globally ordered and locally unordered!
• Order carries useful information• Incorporate order into representation
![Page 6: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/6.jpg)
Matching
Order of features is constrained Sequences may not be of equal lengths Tolerate errors Incorporate the ground distance Measure similarity based on both matched and
unmatched features
],...,,[ 21 mxxx
],...,,[ 21 nyyy
![Page 7: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/7.jpg)
Outline
Introduction Representation based on ordered features Matching criteria
Approach Experiments
Shape retrieval Scene recognition
Conclusion and future work
![Page 8: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/8.jpg)
Approximate String Matching
],...,,[ 21 mxxx
i
icYXd min),(
],...,,[ 21 nyyy
The distance is defined as the minimal cost of operations that transform X into Y
![Page 9: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/9.jpg)
Edit Distance
Three operations Insertion δ (ε, a) Deletion δ (a, ε) Substitution δ (a, b)
Application dependentIntegrates the ground distance
Example: Matching two shapes• Shape Context Descriptor • χ2 distance d(∙,∙)
δ (ε, a) = d ( 0, a ) δ (a, ε) = d ( a, 0 )δ (a, b) = d ( a, b)
2
![Page 10: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/10.jpg)
Edit Distance (cont.)
Metric if each operation cost is also a metric
Preserve the ordering The edit distance reflects
The similarity of the corresponding features by substitution
The dissimilarity of unmatched features by insertion/deletion
Easy to incorporate the ground distance No need to create a visual alphabet
![Page 11: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/11.jpg)
Computing the edit distance
)}.,()1,(
),,(),1(
),,()1,1(min{),(
j
i
ji
yjiD
xjiD
yxjiDjiD
Computation O(mn) Space O(min(m, n))
![Page 12: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/12.jpg)
Alignment of Two Sequences
Cyclic sequences
Search for a pattern X in a duplicate string Y-Yy1 y1yn yn… …
0 0 0 0 0 0 0 0 0 0 0 0 0
x1
xm min { Dm, j }
![Page 13: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/13.jpg)
Outline
Introduction Representation based on ordered features Matching criteria
Approach Experiments
Shape retrieval Scene recognition
Conclusion and future work
![Page 14: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/14.jpg)
Shape Retrieval
MPEG-7 Core Experiment CE-Shape-1 part B 1400 shapes, 70 categories Bull’s eye test# {correct hits in top 40} # {total possible hits}
Representation Uniformly sample 100 points 60-d (5 bins, 12 orientations) shape context
descriptor χ2 distance, δ (ε, a) = δ (a, ε) = 0.5
![Page 15: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/15.jpg)
Shape Retrieval
71.81%
76.51%
82.46%84.29%
0.65
0.7
0.75
0.8
0.85
0.9
1 2 3 4
Same features (shape context descriptor)Different matching methods
Bipartite SC+TPS[Belongie et al.]
COPAP [Scott & Nowak]
Ours
73.51%
84.29% 85.40%
0.65
0.7
0.75
0.8
0.85
0.9
1 2 3
Different features
Fourier Descriptors
Ours(SC+ASM)
IDSC+DP[Ling & Jacobs]
• Find the alignment during matching• No need for any transformation model
![Page 16: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/16.jpg)
Scene Recognition
Scene dataset [Lazebnik et al.] 15 categories, 4485 images, 200-400 images per
category 100 images for training, rest for testing
Representation Spatial pyramid representation Harris-Hessian-Laplace detector + SIFT features 200 visual words
Classification SVM with specified kernel values
![Page 17: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/17.jpg)
Spatial Pyramid Representation
• Level-2 partitioning (16 bags-of-features) achieves the best performance• Each image is represented by 16 bags of features
![Page 18: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/18.jpg)
Scene Recognition
Spatial Pyramid Matching : bag-to-bag matchingOur matching method could allow matching across bags
63.30%
69.21%
66.42%
0.6
0.62
0.64
0.66
0.68
0.7
1 2 3
Using the L-2 partitioning alone and the same ground distance χ2
SPM[Lazebnik et al.]
Bipartite Ours
![Page 19: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/19.jpg)
back
86.74% 76.73% 95.48%
48.71% 78.69% 55.87%
35.21% 67.74% 52.65%
58.52% 73.91% 65.63%
46.55% 53.76% 77.72%
![Page 20: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/20.jpg)
Conclusion
A globally ordered and locally unordered representation for visual data
Approximate String Matching for measuring the similarity between such representations Order is considered Naturally integrates the ground distance between
features Similarity is derived based on both matched and
unmatched features
![Page 21: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/21.jpg)
Future Work
Image registration by using correspondences found by implementing this approach.
Video retrieval and video copy detection by exploring the local alignment ability of string matching approach.
![Page 22: A String Matching Approach for Visual Retrieval and Classification](https://reader036.vdocument.in/reader036/viewer/2022062322/56814937550346895db67b07/html5/thumbnails/22.jpg)
Thank you
More information:http://lbmedia.ece.ucsb.edu