an approximate nearest neighbor retrieval scheme for computationally intensive distance measures
DESCRIPTION
An Approximate Nearest Neighbor Retrieval Scheme for Computationally Intensive Distance Measures. Pratyush Bhatt MS by Research(CVIT). Nearest Neighbor Retrieval. Representation of an object Fixed Length Variable Length (Dis)similarity Function (Distance Measure) Neighborhood - PowerPoint PPT PresentationTRANSCRIPT
An Approximate Nearest Neighbor Retrieval Scheme for
Computationally Intensive Distance Measures
Pratyush BhattMS by Research(CVIT)
Nearest Neighbor Retrieval
• Representation of an object– Fixed Length– Variable Length
• (Dis)similarity Function (Distance Measure)• Neighborhood
• Nearest neighbor retrieval problem can now be formalized as retrieving objects similar to a given object, where similarity is in accordance to a given similarity function
Need of NN retrieval schemes
• If the search space is small, Sequential search is applied for accurate results.
• With increase in memory, volume of data stored online has increased.– Sequential search is time consuming– Need to index data for fast retrieval– Birth of NN search algorithms
Computationally Expensive Distance Measures
• X1 more similar to X2 than X3 visually• L1 distance between X1 and X2 is more than X1 and X3
• Time complexity is super-linear to length of input.
• Edit Distance: O(d2)
Chamfer Distance
Nearest Neighbor Classification
• Computationally expensive distance functions– Nearest neighbor classifiers often impractical for real applications.– Takes over 20 minutes to classify a single object on a modern PC using an
optimized C++ implementation.– Larger the available training data, better will be the accuracy but at the
cost of high computation time.
Motivation(Approx. NN)
• Why to find similarity with all samples when decision is based on top K ?
• How to find top K without finding similarity with all the samples ?
• Solution: Compute K1 > K Approx. NN and find best K in that list.
• In order to compute K1-NN, use fewer explicit matches.
Problem Statement
• Improve nearest neighbor retrieval and classification performance in spaces with computationally expensive distance measures.
• Generate an expansible approximate nearest neighbor list for a given query.
• Dealing with points in non-metric space.
Metric Space
Metric Space
Tree based(KD-tree, R-Tree)
Hashing Based
KD Tree for Metric Space
Non-Metric Space
Wrist Rotation
Applications
• Classification based on K-NN– K is determined empirically
• Identification– Stop when similarity is above a fixed threshold
• Retrieval Applications– Optimizing network usage in peer-to-peer computer networks.– Content-based retrieval systems
• Concept of similarity is abstract– Need to generate expansible list– Learn from user feedbacks
Challenges
• Accuracy depends on the similarity function used to compute Approx. NN
• List can not be pre-computed, should be generated on-the-fly• Should be incremental to support scalability• Non-metric space
– Prohibit application of triangular inequality
Manifold Theory
},.....,,{ 21 nxxxO di Rx
},.....,,{ 21 nyyyE
)( dpR p
• Run MDS on similarity matrix to get
• Embedding of new sample is given by
mean of column of squared matrix requires computation of similarity of new sample to points
t
k
tk
tt
k
vvvL
.....
2
2
1
1#
)(# akLy
nnD
nnD
an
Related Work
• FastMap , Random Reference Objects, Random Line Projections, VP-Trees– Finds embedding of the query by computing only a few exact
distances – Assumption: Triangular Inequality
• BoostMap: Used AdaBoost to combine many simple 1-D embedding
FastMapLipschitz Embedding
Problem Formulation
• Goal : Compute approx. NN of a query point q, from a set S of N points in accordance to similarity function, F.
• Solution : – Split the data into a multi-level hierarchy– Exploit local similarity property to direct the search from top to
bottom
Hierarchical Local Maps(HLM)
1. FIND SIMILARITY OF QUERY WITH POINTS AT TOPMOST LEVEL
2. IDENTIFY NEAREST NEIGHBORS3. GET CHILDREN OF NEAREST NEIGHBORHOW TO FIND SIMILARITY OF QUERY
WITH THESE SAMPLES WITHOUT EXPLICITLY CALCULATING IT ?
4. PROJECT POINTS ON THE MANIFOLD
5. PROJECT QUERY ON THE MANIFOLD
6. FIND NEAREST NEIGHBOURS OF QUERY IN
THIS METRIC SPACE
7. IDENTIFY THESE POINTS IN THE TREE8. TRAVERSE DOWN IN
SAME FASHION
Results
• Unipen Handwriting Database– 15953 digit examples– Divided into training and testing set with 2:1 ratio– Distance Measure : DTW
Number of DTW Computations for K nearest neighbor retrieval
Number of DTW Computations for K nearest neighbor retrieval
Number of DTW Computations for K nearest neighbor retrieval
Classification Accuracy on UNIPEN Dataset using exact and approximate k-NN
Classification Accuracy
K 1 5 10 15 20
DTW 98.1 97.73 97.41 96.99 96.83
HLM 97.65 97.27 97.08 96.69 96.44
Difference 0.45 0.46 0.33 0.30 0.39
• Average No. of DTW computations done by HLM : 160• Expected No. of DTW computations done in brute-force : 5315
Biometric ( Special Case)
• High Inter-Class Variation
• Low Intra-Class Variation
• Low variation in inter-class distances
• Indexing for identification
• How to apply HLM in such cases ??
– Local similarity structure becomes degenerate destroying any manifold structure
Biometric Data
High dimensional data
Relative Contrast :
Iris
• Feature Vector Length 1000• If same class- < 100 bit differ • Else equal probability of each bit to match or not match
– On average 500 bit differ
• Imposter Scores would be around 0.5
Biometric Data
• Such Distribution is bad for Indexing
Softness/Hardness
• Measure of overlap between genuine and imposter classes• Soft Biometric (Face, Body Silhouette)
– Poor Classification Accuracy– Better indexing– Correlates to multi dimensional point
• Hard Biometric (Iris, Fingerprint)– Good Classification Accuracy– Poor indexing– Correlates to high dimensional point
• Need a balance between two
Dataset
• CASIA Iris Image Database V3.0– 855 images corresponding to 285 users in training and testing set– 3 samples per eye
• Simlarity Function– Hamming Distance
– Euclidean Distance (Softer Metric)• Average gray value of a block resulted in 160D feature vector
• Penetration Rate– Percentage of data on which we ran biometric matcher.
• False Reject Rate– Percentage of identification instances in which false rejection occurs
HLM on CASIA Iris Dataset
Synthetic Dataset
• Class center sampled from 1D gaussian (0,1)• Generate d-dimensional by sampling d times• Points of same class sampled from gaussian with mean as
class centers and varying variance.• Total Number of classes: 500• Points in same class
– Training : 10– Testing : 5
Indexing performance varying number of dimensions
Indexing performance varying within class to between class variance ratio
Contributions
• A representation scheme for objects in a dataset that allows for fast retrieval of approximate nearest neighbors in non-euclidean space.
• Search mechanism combined with filter and refine approach is proposed that minimizes the number of exact distance computations for computationally expensive distance measure.
• Study performance of our scheme on biometric data and study the parameters affecting its performance.
Conclusion and Future Work
• Local Similarity Property is well exploited by HLM• Incremental and Scalable• Softer biometric in filtering step combined with hard
biometric in refine step would drastically reduce computation time
• Optimal construction of HLM• Defining a measure for similarity function that allows
hierarchical representation.• Learn a function to find degree of indexibility
– Extract parameters from data distribution and similarity function
Thank You
Related Publication:• Pratyush Bhatt and Anoop Namboodiri “Hierarchical Local
Maps for Robust Approximate Nearest Neighbour Computation” Proceedings of the 7th International Conference on Advances in Pattern Recognition (ICAPR 2009), Feb. 4-6, 2009, Kolkatta, India.