![Page 1: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/1.jpg)
Texture and Material Recognition
2014 February 27
![Page 2: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/2.jpg)
Material RecognitionSimilar to texture recognition.
CURET Texture Database
![Page 3: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/3.jpg)
Flickr Material Database (there are masks)
![Page 4: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/4.jpg)
FMD10 classes (Fabric Foliage Glass Leather Metal Paper Plastic Stone Water Wood)
100 images per class
50 images used for training
![Page 5: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/5.jpg)
Today1. Presenting two papers on local features for texture and material recognition
2. Designed local features
3. Trained to fit a test set by SVM
4. Tested on Flickr Materials Database
![Page 6: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/6.jpg)
First PaperRecognizing Materials Using Perceptually Inspired Features
Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson
IJCV 2013 version of 2010 CVPR paper57.1% on FMD (up from 44.6%)Humans can do 84.9%
![Page 7: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/7.jpg)
Second PaperPairwise Rotation Invariant Co-occurrence Local Binary Pattern
Xianbiao Qi · Rong Xiao · Jun Guo · Lei Zhang
ECCV 201257.4% on FMD
![Page 8: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/8.jpg)
Third PaperToward Robust Material Recognition for Everyday Objects
Diane Hu · Liefeng Bo · Xiaofeng Ren
BMVC 201154% on FMD
![Page 9: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/9.jpg)
Lots of Stuff
First Paper
![Page 10: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/10.jpg)
5 MTurk Human Experiments
![Page 11: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/11.jpg)
Humans2,500 people in MTurk study, 500 per experiment.
84.9% on original images65.3% bilateral filtered64.8% high-pass filtered38.7% synthesized 15x15 (big drop, global to local)46.9% synthesized 30x30
![Page 12: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/12.jpg)
Category matters
![Page 13: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/13.jpg)
Feature GenerationColor and texture features on original images (color, Jet, SIFT)
Edge-based features (slice, ribbon, curvature)
Texture features on bilateral residual (Jet, SIFT)
(Will not cover all these features)
![Page 14: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/14.jpg)
![Page 15: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/15.jpg)
![Page 16: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/16.jpg)
Feature Dictionaryk-means to cluster 8 features into codewords and concatenate all feature codewords into a single dictionary.
![Page 17: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/17.jpg)
Latent Dirichlet Allocation(aLDA method not described here because it doesn’t work)
aLDA vs. SVM 44.6% vs. 57.1%
![Page 18: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/18.jpg)
SVMForm histogram of words and apply binary SVM with histogram intersection. One-versus-all.
![Page 19: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/19.jpg)
Feature ImportanceSIFT, color, curvature, edge-ribbon, micro-SIFT, jet, edge-slice, micro-jet.
SIFT+color can achieve 50.2%
(but, small number of images)
![Page 20: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/20.jpg)
EtcFurther results and conclusions ...
![Page 21: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/21.jpg)
Local Binary Patterns
Second Paper
![Page 22: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/22.jpg)
Local Binary Pattern (Ojala 2002)Invariant to mean value, maps rotation to barrel shifts.
Pattern 11110000 is an edge.
58 patterns <= 2 transitions.
Histogram of 59 or 10 bins
![Page 23: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/23.jpg)
PRI-coLBPIn short:A pair of local binary patterns, gradient aligned at 3 scales.
Trained by SVM/PCA/RBF or SVM/additive kernel approximation
![Page 24: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/24.jpg)
Idea
![Page 25: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/25.jpg)
2D Histogram
Calculate the rotation invariant uniform LBP codeword for point A. Calculate the uniform LBP codeword for point B. Accumulate into 2D histogram.
![Page 26: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/26.jpg)
![Page 27: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/27.jpg)
ExtensionRotation Invariance. Define a local frame at A with local gradient as first axis. Define position of B in this frame.
Evaluate B at three scales and two orientations, along and cross gradient. (edge-ribbon, edge-slice?)
Add RGB, we have 4D histogram. Feature vector has dimension 3*6*10*59 = 10620.
![Page 28: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/28.jpg)
PCA
Feature vector is sparse and can be reduced to 120-150 dimensions (empirically).
![Page 29: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/29.jpg)
Texture Classification
![Page 30: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/30.jpg)
Material Recognition
![Page 31: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/31.jpg)
Oxford Flower
![Page 32: Texture and Material Recognition - Cornell University · Lavanya Sharan · Ce Liu · Ruth Rosenholtz · Edward H. Adelson IJCV 2013 version of 2010 CVPR paper 57.1% on FMD (up from](https://reader034.vdocument.in/reader034/viewer/2022051809/6011fa05bd38190d25740480/html5/thumbnails/32.jpg)
Conclusion1. MIT group got a big boost just by changing their learning method.
2. Pairwise LBP capturing an equivalent quantity of material information?