![Page 1: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/1.jpg)
Towards recognition of everyday objects
Prof. Trevor Darrell
![Page 2: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/2.jpg)
Where we are…
• Great progress in categorical object recognition in computer vision community– advances on CT101, PASCAL, etc.
• Great progress on Robotic sensing, esp., mapping, navigation, etc– SLAM, Grand Challenge, etc
• Yet, broad category-level robotic object recognition in general environments is still nearly nonexistant!
![Page 3: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/3.jpg)
CV: faces and instances
Aachen Cathedral
![Page 4: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/4.jpg)
snaptell.com
![Page 5: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/5.jpg)
like.com
![Page 6: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/6.jpg)
Robotics: Mapping, Navigation…
![Page 7: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/7.jpg)
A modest proposal…• A robot that can recognize / find every object in my
office / kitchen / kid’s playroom?– without requiring a grad student to collect multi-view
training data of each object?
• I don’t think we are on the path to solve this with conventional SIFT + fancy kernels + sup. learning…– even with LabelMe, ImageNet, MechTurk…
• It will likely involve:– multiple sensing modalities (“views”) and semi-supervised
learning (both manifold learning and co-training flavors)
– local features that respect physics of image formation
– active learning at training, and attentive learning at test….
– limited “natural”
interaction with a user
![Page 8: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/8.jpg)
Computer Vision vs
Robotic Vision: Divergent Paradigms?
Computer Vision: – Machine Learning paradigm
– Whoever has the largest dataset wins.
– Leads to least common denominator in terms of features to use
category recognition focus;
weak features…
Robotic Vision: – Sensing paradigm; sensors are
cheap; add them! …3-D sensing, multi-spectral, ultra-
high-res….
– Whoever has the most sensors wins…
– But then we can only get training data from our environment (in situ)!
strong features;
instance recognition focus…
![Page 9: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/9.jpg)
Which one is right?
• Both!
• Neither!
• Key technical problems for next generation robotic visual recognition systems: how to…– bridge category and instance level learning
– fully leverage scene and task context
– simultaneously exploit labeled online data and unlabeled or weakly labeled in-situ data
• Robotics will drive next generation of object recognition challenges….
![Page 10: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/10.jpg)
Rough evolution of visual object recognition research
1970s/80s 2000s1990s 2010+
Get my bag…
?• hierarchical labels• scene/task context• multimodal• interactive• robotic…
![Page 11: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/11.jpg)
• Fusing multiple cues and discovering shared representations across categories…
• Visual Sense Disambiguation…
• Transparent local features…
Recent Progress: Combing Features, Overcoming ambiguity
vs vs
?
features categories
![Page 12: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/12.jpg)
Multiple Cues and Context
![Page 13: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/13.jpg)
Many Local Representations…
Superpixels
[Ren
et al.]
Shape context [Belongie et al.]
Maximally Stable Extremal Regions [Matas et al.]
Geometric Blur [Berg et al.]
SIFT [Lowe]
Salient regions [Kadir et al.]
Harris-Affine [Schmid et al.]
Spin images [Johnson and
Hebert]
Wide variety of proposed local feature representations:
![Page 14: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/14.jpg)
How to Compare Sets of Features?• Each feature type yields set of vectors per image• Unordered, varying number of vectors per instance
?
![Page 15: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/15.jpg)
Pyramid Match
optimal partial matching
for sets with features of dimension
•
Optimal matching
•
Greedy matching
•
Pyramid match
[Grauman and Darrell, ICCV 2005, JMLR 2007]
![Page 16: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/16.jpg)
Cue Combination
• Feature / Kernel combination is very active topic– SVM MKL schemes (Varma, et al.)
– Cross-Validation approaches (recent ICCV papers…)
– Naïve Bayes Nearest Neighbor schemes (Irani)
• Our Gaussian Process formulation has significant efficiency advantage when using 1 vs. all formulation– most of computation is inherently shared across categories
• Good news: significant accuracy improvement!
• Bad news: combination methods all perform about the same, at least on Caltech datasets…– but does help when there are non-informative kernels…
![Page 17: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/17.jpg)
[ See http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-96.html
for details…
]
![Page 18: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/18.jpg)
[Kapoor, Urtasun, Grauman, Darrell, IJCV, to appear]
![Page 19: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/19.jpg)
Combining 2D and 3D
• How to best exploit 3-D sensors?– very hard to get good local shape estimates
• Exploit 3-D sensing as context for 2-D reco.
• 3-D scene context– 3D can provide summary of overall environment
– find support surfaces
• Estimate and exploit absolute size
constraints– model size variation of overall object and
local
patches based on training data or external knowledge
![Page 20: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/20.jpg)
Indoor Support surfaces
Table surface extraction from 3-D scene data:
![Page 21: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/21.jpg)
Indoor search constraints
• Use surface
and size constraints…
![Page 22: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/22.jpg)
Notion of Absolute Size
• Absolute feature size
• Absolute object size
Vs.
Vs.
![Page 23: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/23.jpg)
Branch & Bound for Fast Detection
[Lampert]
•Feature computation•Codebook matching•Discriminative weights•Sliding window•Detections
![Page 24: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/24.jpg)
Object Detection
• Exhaustive search is costly –
in particular
for 3d• Previous work on
branch&bound provides speed up for 2d
• Use upper bounds on bounding box intervals
A
l
l
p
o
s
s
i
b
l
2-42-4 3-53-5
2-2.52-2.5 3-43-4X
![Page 25: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/25.jpg)
Current Detection Demo Results
![Page 26: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/26.jpg)
Joint Category Learning
![Page 27: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/27.jpg)
Standard “1 vs. all”
paradigm….
SVM/GPC – Category 1
SVM/GPC – Category 2
SVM/GPC – Category 3
SVM/GPC – Category 4
SVM/GPC – Category 256
…
SVM/GPC – Category 10,000?
How to exploit shared structure?
![Page 28: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/28.jpg)
Consider ensemble of classifiers
SVM/GPC – Category 1 w1
SVM/GPC – Category 2 w2
SVM/GPC – Category 3 w3
SVM/GPC – Category 4 w4
SVM/GPC – Category 256
…
SVM/GPC – Category w10,000
classifier weights
![Page 29: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/29.jpg)
Consider ensemble of classifiers
SVM/GPC – Category 1 w1
SVM/GPC – Category 2 w2
SVM/GPC – Category 256
…
SVM/GPC – Category wn
W = [ w1
w2
… wn
]
Related tasks and/or object part structure will lead to correlated patterns in W…[Quattoni, Collins, Darrell, CVPR
2007] explore Ando+Zhang style structure learning for scene recognition tasks.
Learn W jointly? [Quattoni, Collins, Darrell, CVPR
2008] explore joint spare optimization via matrix norm penalty.
[Quattoni, Carreras, Collins, Darrell, ICML 2009] report an efficient learning scheme for this approach…
![Page 30: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/30.jpg)
Joint Sparse Approximation
xwxf )(
Dyx
d
jjwCyxfl
),( 1||)),((min
w
• Consider learning a single sparse linear classifier of the form:
That is, we want only a few features with non-zero coefficients
• L1 regularization well-known to yield sparse solutions:
Classificationerror
L1 penalizesnon-sparse solutions
![Page 31: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/31.jpg)
Joint Sparse Approximation
m
kmk
Dyxk
CyxflD
k121
),(,...,, ),....,,R()),((
||1min www
m21 www
Optimization over several tasks jointly:
Average Losson training set k
penalizes solutions that
utilize too many features
xxf kk w)(
Key idea: use a matrix norm…[Obozinski et al. 2006, Argyriou et al. 2006, Amit et al. 2007 ]
![Page 32: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/32.jpg)
Joint Regularization Penalty
rowszerononW #)R(
mddd
m
m
WWW
WWWWWW
,2,1,
,22,21,2
,12,11,1
How do we penalize solutions that use too many features?
Coefficients forfor feature 2
Coefficients forclassifier 2
Would lead to a hard combinatorial problem .
![Page 33: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/33.jpg)
Joint Regularization PenaltyWe use a L1-∞
norm [Tropp 2006]
d
iikk
WW1
|)(|max)R(
The combination of the two norms results in a solution where only a few features are used but the features used will contribute in solving many classification problems.
This norm combines:
An L1 norm on the maximum absolute values of the coefficients across tasks promotes sparsity.
Use few features
The L∞
norm on each row promotes non- sparsity on the rows. Share
features
![Page 34: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/34.jpg)
Joint Sparse Approximation
m
k
d
iikkk
Dyxk
WCyxflD
k1 1),(|)(|max)),((
||1minW
Using the L1-∞
norm we can rewrite our objective function as:
For any convex loss this is a convex objective.
For the hinge loss the optimization problem can be expressed as a linear program. [Quattoni et al. CVPR 2008]
See also [Quattoni et al ICML 2009] for efficient large scale solutions.
![Page 35: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/35.jpg)
News Image Classification Experiments
15 30 60 120 2400.32
0.34
0.36
0.38
0.4
0.42
0.44
0.46
# training examples per task
Mea
n E
ER
Reuters Dataset Results
L2L1L1-INF
Absolute Weights L1
Feat
ure
5 10 15 20 25 30 35 40
500
1000
1500
2000
2500
3000
10
20
30
40
50
60
Absolute Weights L1-INF
Fea
ture
task
5 10 15 20 25 30 35 40
500
1000
1500
2000
2500
3000
0.01
0.02
0.03
0.04
0.05
0.06
SuperBowlDanish
CartoonsSharonAustralian
openTrapped
coal miners
Goldenglobes Grammys Figure
skating AcademyAwards Iraq
L1,∞L1
![Page 36: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/36.jpg)
Visual Sense Disambiguation
![Page 37: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/37.jpg)
38
Goal: Object recognition in situated environments
•
Imagine using natural dialogue to instantiate object models in a robot
That’s a cat over there…
This is one of my purses.
There’s a lamp…
![Page 38: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/38.jpg)
39
Speech, image can be complementary…
a pan...
That’s a pen!
Copy machine..
ant →
fanface →
basspiano →
cannon
![Page 39: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/39.jpg)
40
Using image to aid speech recognition
Objectrecognition
![Page 40: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/40.jpg)
41
Experiments on Caltech101•
Asked users to speak the object name, added noise
•
Showed benefit from fusion at all noise levels
ant →
fanface →
basspiano →
cannon
![Page 41: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/41.jpg)
42
Large Object Vocabulary?
Objectrecognition
![Page 42: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/42.jpg)
43
Large Object Vocabulary?
![Page 43: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/43.jpg)
44
Category Discovery: “Watch”…
![Page 44: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/44.jpg)
45
Sources of visual polysemy
Would rather watch… Suicide watch
Hurricane, tornado watch
Watch out!
Celebrity watch
![Page 45: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/45.jpg)
46
Taking advantage of text contexts
icrystal rfid wrist watch features watch masterpiece innovative watch making craftsmanship absolute precision fine charm high scratch resistance anti-
allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch
solar powered available watch ticket key purse identity card special offer place order rfid wrist watch absolutely free rfid watch black
wrist strap rfid watch orange wrist strap rfid watch stainless steel privacy
disclaimer copyright icrystal pty website
![Page 46: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/46.jpg)
47
Dictionary model•
Use entry text to learn a probability distribution over words for that sense
•
Problem: entries contain very little text–
Expand by adding synonyms, hyponyms, 1st-level hypernyms–
Still, very few words are covered!
•S: (n) mouse (any of numerous small rodents typically resembling diminutive rats having pointed snouts and small ears on elongated bodies with slender usually hairless tails) •direct hyponym / full hyponym
•S: (n) house mouse, Mus musculus (brownish-grey Old World mouse now a common household pest worldwide) •S: (n) harvest mouse, Micromyx minutus (small reddish-brown Eurasian mouse inhabiting e.g. cornfields) •S: (n) field mouse, fieldmouse (any nocturnal Old World mouse of the genus Apodemus inhabiting woods and fields and gardens) •S: (n) nude mouse (a mouse with a genetic defect that prevents them from growing hair and also prevents them from immunologically rejecting human cells and tissues; widely used in preclinical trials) •S: (n) wood mouse (any of various New World woodland mice)
•direct hypernym / inherited hypernym / sister term•S: (n) rodent, gnawer (relatively small placental mammals having a single pair of constantly growing incisor teeth specialized for gnawing)
![Page 47: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/47.jpg)
48
Topic space
Idea:
use large collection of unlabeled text
to learn hidden topics which align with different senses/uses of the word
live mice petrodent ear price pest
house bait old animal human tube need tail species gene head breed body love color care
friend wood cat weight white water
. . .
![Page 48: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/48.jpg)
49
Learning visual senses: overview
Search Engine Watch Search Engine Watch
is the authoritative guide to search engine marketing (SEM) and search engine optimization (SEO), offering the latest news about search ...
searchenginewatch.com/ -
38k -
Cached - Similar pages - Note this watch - MDCWatches
for assignment to a property named prop in this object, calling handler(prop, oldval, newval) whenever prop is set and storing the return value in ...
developer.mozilla.org/en/Core_Java
Script_1.5_Reference/Global_Objec
ts/Object/watch - 30k - Cached - Similar pages - Note this
Search Engine Watch Search Engine Watch
is the authoritative guide to search engine marketing (SEM) and search engine optimization (SEO), offering the latest news about search ...
searchenginewatch.com/ -
38k -
Cached - Similar pages - Note this watch - MDC
Watches
for assignment to a property named prop in this object, calling handler(prop, oldval, newval) whenever prop is set and storing the return value in ...
developer.mozilla.org/en/Core_Java
Script_1.5_Reference/Global_Objec
ts/Object/watch - 30k - Cached - Similar pages - Note this
dictionary definitions
latent topic space
unlabeled text
dictionary model P( sense | page)
unlabeled images+text training images
visual senseclassifier
![Page 49: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/49.jpg)
50
Clustering + Word Sense Model
Object Sense: drinking container
Abstract Sense: sporting event
Object Sense: loving cup (trophy)
Search Word: “cup”
Online Dictionary
Word to search for:Noun
cup Search Dictionary
• cup (a small open container usually used for drinking; usually has a handle) "he put the cup back in the saucer"; "the handle of the cup was missing"
• cup, loving cup (a large metal vessel with two handles that is awarded as a trophy to the winner of a competition) "the school kept the cups is a special glass case”
• a major sporting event or competition “the world cup”, “the Stanley cup”
Filtering Abstract Senses
![Page 50: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/50.jpg)
51
Concrete vs. abstract senses
Mouse: Noun•<noun.animal>S: (n) mouse (any of numerous small rodents typically resembling diminutive rats having pointed snouts and small ears on elongated bodies with slender usually hairless tails) •<noun.state>S: (n) shiner, black eye, mouse (a swollen bruise caused by a blow to the eye) •<noun.person>S: (n) mouse (person who is quiet or timid) •<noun.artifact>S: (n) mouse, computer mouse (a hand-operated electronic device that controls the coordinates of a cursor on your computer screen as you move it around on a pad; on the bottom of the device is a ball that rolls on the surface of the pad) "a mouse takes much more room than a trackball"
• How can we determine if a sense is concrete or abstract?– Use a natural language processing method to learn classifier– Use existing dictionary information: e.g. WordNet’s lexical file tags
![Page 51: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/51.jpg)
52
Filtering visual senses
Yahoo Search: “fork” DICTIONARY
1: (n) fork
(cutlery used for serving and eating food) 2: (n) branching, ramification, fork, forking (the act of branching out or dividing into branches) 3: (n) fork, crotch (the region of the angle formed by the junction of two branches) "they took the south fork"; "he climbed into the crotch of a tree"4: (n) fork
(an agricultural tool used for lifting or digging; has a handle and metal prongs) 5: (n) crotch, fork
(the angle formed by the inner sides of the legs where they join the human trunk)
![Page 52: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/52.jpg)
53
Filtering visual senses
Artifact sense of “fork” DICTIONARY
1: (n) fork
(cutlery used for serving and eating food) 2: (n) branching, ramification, fork, forking (the act of branching out or dividing into branches) 3: (n) fork, crotch (the region of the angle formed by the junction of two branches) "they took the south fork"; "he climbed into the crotch of a tree"4: (n) fork
(an agricultural tool used for lifting or digging; has a handle and metal prongs) 5: (n) crotch, fork
(the angle formed by the inner sides of the legs where they join the human trunk)
![Page 53: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/53.jpg)
54
Filtering visual senses
Yahoo Search: “telephone” DICTIONARY
1: (n) telephone, phone, telephone set (electronic equipment that converts sound into electrical signals that can be transmitted over distances and then converts received signals back into sounds)
2: (n) telephone, telephony (transmitting speech at a distance)
![Page 54: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/54.jpg)
55
Filtering visual senses
Artifact sense: “telephone” DICTIONARY
1: (n) telephone, phone, telephone set (electronic equipment that converts sound into electrical signals that can be transmitted over distances and then converts received signals back into sounds)
2: (n) telephone, telephony (transmitting speech at a distance)
![Page 55: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/55.jpg)
56
Topic adaptation•
Original LDA topics are learned on text-only unlabeled data
•
Adapt to image-text data via semi-supervised Gibbs sampling
•
E.g.: one of “fork” topics:
product bike null tool tube seal set price oil knife spoon spring
ship use item accessory handle shop order
remove store custom home weight steel supply cap clamp fit
false . . .
cutlery knife spoon product set price handle
steel tool item stainless null bike tube seal oil knive
kitchen utensil ship order use table sp ring supply design piece carve weight shop
. . .
![Page 56: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/56.jpg)
57
“fork”: using original topics
fork liftroad forkbike fork
etc.
![Page 57: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/57.jpg)
58
“fork”: using adapted topics
knifespooncutlery
bike forketc.
![Page 58: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/58.jpg)
59
Work in progress…
•
Showed that combining speech and image classifiers improves object reference resolution
•
Proposed an unsupervised method to learn sense-specific object models from web text and image data
•
Integrated large scale demo forthcoming
•
See Kate’s NIPS 2009 & 2008 papers…
![Page 59: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/59.jpg)
Transparent Local Features
![Page 60: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/60.jpg)
![Page 61: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/61.jpg)
Dealing with Transparency
![Page 62: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/62.jpg)
Motivation
•
Transparent objects made out of glass or plastic are ubiquitous in domestic environments
•
Traditional local feature approach inappropriate
•
Full physical model intractable
![Page 63: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/63.jpg)
Local Additive Feature Model
•
Significant variation in patch appearance
•
... but common latent structure
![Page 64: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/64.jpg)
new LDA-SIFT model
![Page 65: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/65.jpg)
LDA-SIFT
![Page 66: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/66.jpg)
Transparent Visual Words
•
For each patch we infer the latent mixture activations that characterize the additive structure
•
We model the glass by learning a spatial layout of discrete “transparent local feature”
activations
![Page 67: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/67.jpg)
Transparent Visual Words
Latent component Average occurrenceon train Occurrences on test
![Page 68: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/68.jpg)
Recognition Architecture
LDA
XY
T
Classifier
glas
sba
ckgr
ound
X
T
…
…
Y
![Page 69: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/69.jpg)
Results: general vocabulary
• Training on 4 different glasses in front of screen
• Testing on 49 glass instances in home environment
• Sliding window linear SVM-detection
![Page 70: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/70.jpg)
Recognition Architecture
sLDA
XY
T
Classifier
glas
sba
ckgr
ound
X
T
…
…
Y
![Page 71: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/71.jpg)
Results: sLDA
• Training on 4 different glasses in front of screen
• Testing on 49 glass instances in home environment
• Sliding window linear SVM-detection
![Page 72: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/72.jpg)
Conclusion
•
Traditional local feature models (VQ, NN) are poorly suited for transparent object recognition
•
Proposed additive local feature models can detect superimposed image structures
•
Developed statistical approach to learn such representations using probabilistic topic models
•
Sparse factorization of local gradient statistics•
Encouraging results on real-world data
![Page 73: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/73.jpg)
Future Work
•
Different feature representations; extend model in hierarchical fashion
•
Investigate addition of material property cues; discriminative inverse local light transport models
•
Explore benefits for opaque object recognition; understand relationship to sparse image coding as well as to biological motivated models
![Page 74: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/74.jpg)
• Fusing multiple cues and discovering shared representations across categories…
• Visual Sense Disambiguation…
• Transparent local features…
Recent Progress: Combing Features, Overcoming ambiguity
vs vs
?
features categories
![Page 75: Towards recognition of everyday objectscharm high scratch resistance anti- allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar](https://reader033.vdocument.in/reader033/viewer/2022060301/5f0858df7e708231d4218f47/html5/thumbnails/75.jpg)
For more information…
•
Probabilistic multi-kernel fusion–
Christhoudias, Urtasun, Darrell, CVPR 2009
•
Joint regularization across categories–
Quattoni, Carreras, Collins, Darrell,
ICML 2009.
•
Machine Learning for Multimodal sense grounding–
Saenko
and Darrell, NIPS 2008, NIPS 2009
•
Local feature models for transparent objects–
Fritz, Bradski, Black, and Darrell, NIPS 2009