computer vision group university of california berkeley shape matching and object recognition using...
Post on 19-Dec-2015
219 views
TRANSCRIPT
![Page 1: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/1.jpg)
Computer Vision GroupUniversity of California Berkeley
Shape Matching and Object Recognition using Shape Contexts
Jitendra Malik
U.C. Berkeley
(joint work with S. Belongie, J. Puzicha, G. Mori)
![Page 2: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/2.jpg)
Computer Vision GroupUniversity of California Berkeley
Outline
• Shape matching and isolated object recognition
• Scaling up to general object recognition
![Page 3: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/3.jpg)
Computer Vision GroupUniversity of California Berkeley
Biological Shape
• D’Arcy Thompson: On Growth and Form, 1917– studied transformations between shapes of organisms
![Page 4: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/4.jpg)
Computer Vision GroupUniversity of California Berkeley
Deformable Templates: Related Work
• Fischler & Elschlager (1973)
• Grenander et al. (1991)
• Yuille (1991)
• von der Malsburg (1993)
![Page 5: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/5.jpg)
Computer Vision GroupUniversity of California Berkeley
Matching Framework
• Find correspondences between points on shape
• Estimate transformation
• Measure similarity
model target
...
![Page 6: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/6.jpg)
Computer Vision GroupUniversity of California Berkeley
Comparing Pointsets
![Page 7: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/7.jpg)
Computer Vision GroupUniversity of California Berkeley
Shape ContextCount the number of points inside each bin, e.g.:
Count = 4
Count = 10
...
Compact representation of distribution of points relative to each point
![Page 8: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/8.jpg)
Computer Vision GroupUniversity of California Berkeley
Shape Context
![Page 9: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/9.jpg)
Computer Vision GroupUniversity of California Berkeley
Shape Contexts
• Invariant under translation and scale
• Can be made invariant to rotation by using local tangent orientation frame
• Tolerant to small affine distortion– Log-polar bins make spatial blur proportional to r
Cf. Spin Images (Johnson & Hebert) - range image registration
![Page 10: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/10.jpg)
Computer Vision GroupUniversity of California Berkeley
Comparing Shape Contexts
Compute matching costs using Chi Squared distance:
Recover correspondences by solving linear assignment problem with costs Cij
[Jonker & Volgenant 1987]
![Page 11: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/11.jpg)
Computer Vision GroupUniversity of California Berkeley
Matching Framework
• Find correspondences between points on shape
• Estimate transformation
• Measure similarity
model target
...
![Page 12: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/12.jpg)
Computer Vision GroupUniversity of California Berkeley
• 2D counterpart to cubic spline:
• Minimizes bending energy:
• Solve by inverting linear system
• Can be regularized when data is inexact
Thin Plate Spline Model
Duchon (1977), Meinguet (1979), Wahba (1991)
![Page 13: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/13.jpg)
Computer Vision GroupUniversity of California Berkeley
MatchingExample
model target
![Page 14: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/14.jpg)
Computer Vision GroupUniversity of California Berkeley
Outlier Test Example
![Page 15: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/15.jpg)
Computer Vision GroupUniversity of California Berkeley
Synthetic Test Results
Fish - deformation + noise Fish - deformation + outliers
ICP Shape Context Chui & Rangarajan
![Page 16: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/16.jpg)
Computer Vision GroupUniversity of California Berkeley
Matching Framework
• Find correspondences between points on shape
• Estimate transformation
• Measure similarity
model target
...
![Page 17: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/17.jpg)
Computer Vision GroupUniversity of California Berkeley
Terms in Similarity Score• Shape Context difference
• Local Image appearance difference– orientation– gray-level correlation in Gaussian window– … (many more possible)
• Bending energy
![Page 18: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/18.jpg)
Computer Vision GroupUniversity of California Berkeley
Object Recognition Experiments
• Kimia silhouette dataset
• Handwritten digits
• COIL 3D objects (Nayar-Murase)
• Human body configurations
• Trademarks
![Page 19: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/19.jpg)
Computer Vision GroupUniversity of California Berkeley
Shape Similarity: Kimia dataset
![Page 20: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/20.jpg)
Computer Vision GroupUniversity of California Berkeley
Quantitative Comparison
rank
Num
ber
corr
ect
![Page 21: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/21.jpg)
Computer Vision GroupUniversity of California Berkeley
Handwritten Digit Recognition
• MNIST 60 000: – linear: 12.0%
– 40 PCA+ quad: 3.3%
– 1000 RBF +linear: 3.6%
– K-NN: 5%
– K-NN (deskewed): 2.4%
– K-NN (tangent dist.): 1.1%
– SVM: 1.1%
– LeNet 5: 0.95%
• MNIST 600 000 (distortions): – LeNet 5: 0.8%– SVM: 0.8%– Boosted LeNet 4: 0.7%
• MNIST 20 000: – K-NN, Shape Context
matching: 0.63%
![Page 22: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/22.jpg)
Computer Vision GroupUniversity of California Berkeley
![Page 23: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/23.jpg)
Computer Vision GroupUniversity of California Berkeley
Results: Digit Recognition
1-NN classifier using:Shape context + 0.3 * bending + 1.6 * image appearance
![Page 24: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/24.jpg)
Computer Vision GroupUniversity of California Berkeley
COIL Object Database
![Page 25: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/25.jpg)
Computer Vision GroupUniversity of California Berkeley
Error vs. Number of Views
![Page 26: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/26.jpg)
Computer Vision GroupUniversity of California Berkeley
Prototypes Selected for 2 Categories
Details in Belongie, Malik & Puzicha (NIPS2000)
![Page 27: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/27.jpg)
Computer Vision GroupUniversity of California Berkeley
Editing: K-medoids
• Input: similarity matrix
• Select: K prototypes
• Minimize: mean distance to nearest prototype
• Algorithm: – iterative– split cluster with most errors
• Result: Adaptive distribution of resources (cfr. aspect graphs)
![Page 28: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/28.jpg)
Computer Vision GroupUniversity of California Berkeley
Error vs. Number of Views
![Page 29: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/29.jpg)
Computer Vision GroupUniversity of California Berkeley
Human body configurations
![Page 30: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/30.jpg)
Computer Vision GroupUniversity of California Berkeley
Automatically Locating Keypoints
• User marks keypoints on exemplars
• Find correspondence with test shape
• Transfer keypoint position from exemplar to the test shape.
![Page 31: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/31.jpg)
Computer Vision GroupUniversity of California Berkeley
Results
![Page 32: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/32.jpg)
Computer Vision GroupUniversity of California Berkeley
Trademark Similarity
![Page 33: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/33.jpg)
Computer Vision GroupUniversity of California Berkeley
Outline
• Shape matching and isolated object recognition
• Scaling up to general object recognition– Many objects (Mori, Belongie & Malik, CVPR 01)– Gray scale matching (Berg & Malik, CVPR 01)– Objects in scenes (scanning or segmentation)
![Page 34: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/34.jpg)
Computer Vision GroupUniversity of California Berkeley
Mori, Belongie, Malik (CVPR 01)
• Fast Pruning– Given a query shape, quickly return a shortlist of
candidate matches– Database of known objects will be large: ~30000
• Detailed Matching– Perform computationally expensive comparisons
on only the few shapes in the shortlist
![Page 35: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/35.jpg)
Computer Vision GroupUniversity of California Berkeley
Representative Shape Contexts
• Match using only a few shape contexts– Don’t need to
compare every one
),(minarg
),(),(
2*
*
1
2
ui
jqueryui
ij
query
r
jiquery
SCSCSC
SCSCSSdist
![Page 36: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/36.jpg)
Computer Vision GroupUniversity of California Berkeley
Snodgrass Results
![Page 37: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/37.jpg)
Computer Vision GroupUniversity of California Berkeley
Results
![Page 38: Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint](https://reader036.vdocument.in/reader036/viewer/2022062714/56649d2a5503460f949fefe1/html5/thumbnails/38.jpg)
Computer Vision GroupUniversity of California Berkeley
Conclusion
• Introduced new matching algorithm matching based on shape contexts and TPS
• Robust to outliers & noise
• Forms basis of object recognition technique that performs well in a variety of domains using exactly the same algorithm