searching for category-consistent features: a computational...
TRANSCRIPT
![Page 1: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/1.jpg)
Searching for Category-Consistent Features: A Computational Approach to Understanding Visual Category Representation
CHEN - PI NG YU, JUST I N MAXFI ELD, AND G REG ORY J ZEL I NSKY
CATEGORY-CONSISTENT FEATURES 1
![Page 2: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/2.jpg)
Hierarchical Levels
Mammal • Superordinate
Dog • Basic
Golden • Subordinate Retriever
CATEGORY-CONSISTENT FEATURES 2
![Page 3: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/3.jpg)
The Questions 1. What might affect the performance of categorical search? ◦ Our hypothesis: the specificity and distinctiveness of the category.
◦ Specificity and distinctiveness are quantified by categorical visual features.
2. How might the visual features of object categories be extracted? ◦ Our answer: learn a feature representation for each object category.
3. How likely is this hypothesis to be true? ◦ Collect behavioral data on categorical search performance
◦ Build the model, and learn the generative features from the data
◦ Evaluate the model’s fit against the behavioral data
CATEGORY-CONSISTENT FEATURES 3
![Page 4: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/4.jpg)
Specificity & Distinctiveness Subordinate level: ◦ Very Specific
◦ Not Distinctive
Basic Level: ◦ Somewhat Specific
◦ Somewhat Distinctive
Superordinate Level: ◦ Not at all specific
◦ Very Distinctive
Subordinate Basic Superordinate
SpecificityDistinctiveness
CATEGORY-CONSISTENT FEATURES 4
i.e. Taxis Cars Vehicles
![Page 5: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/5.jpg)
Search Procedure
Vehicle
Plane
Passenger Airliner
+
2500 ms 500 ms Search Display
(guidance epoch) Search Display
(verification epoch)
CATEGORY-CONSISTENT FEATURES 5
26 subjects, 288 trials (target present + absent)
16°
![Page 6: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/6.jpg)
Time to Target
550
600
650
700
750
800
Subordinate Basic Superordinate
Time to Target
(ms)
Cue
CATEGORY-CONSISTENT FEATURES 6
![Page 7: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/7.jpg)
Target Fixated First
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Subordinate BasicSuperordinate
Proportion of
Immediate Target
Fixations
Cue
CATEGORY-CONSISTENT FEATURES 7
![Page 8: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/8.jpg)
Verification Time
600
700
800
900
1000
1100
1200
1300
1400
Subordinate Basic Superordinate
Verification Time (ms)
Cue
CATEGORY-CONSISTENT FEATURES 8
![Page 9: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/9.jpg)
Model - Feature representation Learning a novel object category:
Finding the commonalities that represent the category.
CATEGORY-CONSISTENT FEATURES 9
i.e. What is a dragon fruit? ◦ Ellipsoid, pinkish red,
smooth texture, extruding green pedals.
A generative model: Category-Consistent-Features (CCFs).
![Page 10: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/10.jpg)
Discriminative vs Generative
CATEGORY-CONSISTENT FEATURES 10
![Page 11: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/11.jpg)
CATEGORY-CONSISTENT FEATURES 11
Superordinate Basic Subordinate
Vehicle Car Police Car
Taxi
Race Car
Boat Sail Boat
Cruise Ship
Speed Boat
Plane Passenger Airliner
Biplane
Fighter Jet
Truck 18 Wheeler
Fire Truck
Pickup Truck
Furniture Cabinet Kitchen Cabinet
Filing Cabinet
China Cabinet
Chair Folding Chair
Office Chair
Dining Room Chair
Bed Twin Bed
Canopy Bed
Bunk Bed
Table Coffee Table
Dining Room Table
End Table
Clothing Pants Jeans
Dress Pants
Pajama Pants
Shirt Dress Shirt
T-shirt
Long Sleeve Shirt
Hat Baseball Hat
Knit Cap
Cowboy Hat
Jacket Winter Jacket
Windbreaker
Trench Coat
Dessert Ice Cream Chocolate Ice Cream
Mint Choc. Chip Ice Cream
Strawberry Ice Cream
Pie Pecan Pie
Blueberry Pie
Lemon Meringue Pie
![Page 12: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/12.jpg)
![Page 13: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/13.jpg)
![Page 14: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/14.jpg)
![Page 15: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/15.jpg)
Category-Consistent Feature Model
CATEGORY-CONSISTENT FEATURES 15
![Page 16: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/16.jpg)
Category-Consistent Features
Bag-of-Words Method
00
Step 1: Extract features and create a visual dictionary
visual words in dictionary
Step 2: Create descriptors in this common feature space for individual exemplars
bag-of-words histogram
Figure adapted from Bandara (2014)
CATEGORY-CONSISTENT FEATURES 16
![Page 17: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/17.jpg)
Taxis Cars Vehicles
CATEGORY-CONSISTENT FEATURES 17
![Page 18: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/18.jpg)
Taxis Cars Vehicles
CATEGORY-CONSISTENT FEATURES 18
![Page 19: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/19.jpg)
Histogram Visualization Taxis
Cars
Visual Words
Exe
mp
lars
1
1064
1
100
Exe
mp
lars
300
1
Visual Words 1064 1
CATEGORY-CONSISTENT FEATURES 19
![Page 20: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/20.jpg)
Histogram Visualization Taxis
Cars
Visual Words
Exe
mp
lars
1
1064
1
100
Exe
mp
lars
300
1
Visual Words 1064 1
CATEGORY-CONSISTENT FEATURES 20
![Page 21: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/21.jpg)
Histogram Visualization Taxis
Cars
Visual Words
Exe
mp
lars
1
1064
1
100
Exe
mp
lars
300
1
Visual Words 1064 1
CATEGORY-CONSISTENT FEATURES 21
![Page 22: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/22.jpg)
CATEGORY-CONSISTENT FEATURES 22
Taxis Cars Vehicles
![Page 23: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/23.jpg)
Taxis Cars Vehicles
CATEGORY-CONSISTENT FEATURES 23
What are the representative features (CCFs)?
High frequency, low variation
![Page 24: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/24.jpg)
CATEGORY-CONSISTENT FEATURES 24
Taxis
![Page 25: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/25.jpg)
CATEGORY-CONSISTENT FEATURES 25
Taxis
![Page 26: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/26.jpg)
Interq
uartile R
ange R
ule
Inverse C
oefficien
t of V
ariation
CATEGORY-CONSISTENT FEATURES 26
Taxis Category-Consistent Features
![Page 27: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/27.jpg)
Histogram Visualization Taxis
CATEGORY-CONSISTENT FEATURES 27
Dress pants
![Page 28: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/28.jpg)
Visualized CCFs Knit caps
Sugar cookie
Sailboats
CATEGORY-CONSISTENT FEATURES 28
![Page 29: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/29.jpg)
Number of CCFs = Specificity
Number of CCFs was highest at the subordinate level, approximating the specific within category similarity
What about between category distinctiveness?
0.45
0.5
0.55
0.6
0.65
55
60
65
70
75
80
85
Subordinate Basic Superordinate
Nu
mb
er
of
CC
Fs
CATEGORY-CONSISTENT FEATURES 29
![Page 30: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/30.jpg)
Sibling Distance = Distinctiveness
boats
sailboat
cars
police car race car
CATEGORY-CONSISTENT FEATURES 30
Siblings: categories that share the same parent.
![Page 31: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/31.jpg)
Specificity & Distinctiveness
Nu
mb
er
of
CC
Fs
Me
an
Sib
lin
g D
ista
nce
0.45
0.5
0.55
0.6
0.65
55
60
65
70
75
80
85
Subordinate Basic Superordinate
CATEGORY-CONSISTENT FEATURES 31
Subordinate level: ◦ Very Specific
◦ Not Distinctive
Basic Level: ◦ Somewhat Specific
◦ Somewhat Distinctive
Superordinate Level: ◦ Not at all specific
◦ Very Distinctive
![Page 32: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/32.jpg)
Model Performance
CATEGORY-CONSISTENT FEATURES 32
550
600
650
700
750
800
850
900
Subordinate Basic Superordinate
Tim
e t
o T
arg
et
(ms)
Level in Category Hierarchy
Control
Behavioral
Model
800
900
1000
1100
1200
1300
1400
Subordinate Basic Superordinate
Ve
rifi
cati
on
Tim
e (
ms)
Level in Category Hierarchy
Control
Behavioral
Model
Guidance: #-of-CCFs Verification: #CCFs*Sibling-Dist
![Page 33: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/33.jpg)
Trial-by-Trial fit
CATEGORY-CONSISTENT FEATURES 33
CCF model vs Subject Model (144 target present trials)
Paired t-test: correlations were not significantly different, other than the superordinate level (random first-target-fixated).
Psychological Science, in press 2016
![Page 34: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/34.jpg)
Current work
CATEGORY-CONSISTENT FEATURES 34
How can we do even better? ◦ Predict categorical search performance on individual categories.
Drawbacks of the BoW-CCF model ◦ Single level of image features
◦ Hand designed features (SIFT)
Convolutional Neural Network (CNN-CCF) ◦ Hierarchical features
◦ Features learned directly from images
![Page 35: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/35.jpg)
Convolutional Neural Networks (CNNs)
CATEGORY-CONSISTENT FEATURES 35
A typical neural network A Convolutional Neural Network
Searching for pulsars using image pattern recognition - Zhu, W.W. et al. Astrophys.J. 781 (2014) 2, 117 arXiv:1309.0776
![Page 36: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/36.jpg)
CNN features
CATEGORY-CONSISTENT FEATURES 36
https://devblogs.nvidia.com/parallelforall/accelerate-machine-learning-cudnn-deep-neural-network-library/
http://cs231n.github.io/convolutional-networks/
![Page 37: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/37.jpg)
Ventral-stream CNN-CCF
CATEGORY-CONSISTENT FEATURES 37
AlexNet, NIPS 2012
96 (11) 256 (51) 384 (99) 384 (131) 256 (163)
Kravitz et al. 2012
Ventral-stream CNN
442 (11) 470 (16) 213 (53) 154 (64) 71 (132)
Layer sizes are based on Felleman et al. 1991
![Page 38: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/38.jpg)
Ventral-stream CNN-CCF
CATEGORY-CONSISTENT FEATURES 38
Convolutional layers FC layers
vsCNN-CCFs: the filters that are highly, and consistently activated, given images of a category.
Goal: search performance prediction for individual categories
![Page 39: Searching for Category-Consistent Features: A Computational …chenpingyu.org/docs/ccf_presentation2016.pdf · Searching for Category-Consistent Features: A Computational Approach](https://reader036.vdocument.in/reader036/viewer/2022081405/5f0d05737e708231d438496d/html5/thumbnails/39.jpg)
Acknowledgments Justin Maxfield and Greg Zelinsky
Hossein Adeli & Eye Cog Lab RA’s
NSF Grants IIS-1111047 & IIS-1161876
CATEGORY-CONSISTENT FEATURES 39
E Y E C O G L A B