what helps where – and why? semantic relatedness for knowledge transfer
DESCRIPTION
What Helps Where – And Why? Semantic Relatedness for Knowledge Transfer. Marcus Rohrbach 1,2 Michael Stark 1,2 György Szarvas 1 Iryna Gurevych 1 Bernt Schiele 1,2 1 Department of Computer Science, TU Darmstadt 2 MPI Informatics, Saarbrücken. - PowerPoint PPT PresentationTRANSCRIPT
What Helps Where – And Why?Semantic Relatedness for Knowledge Transfer
Marcus Rohrbach1,2 Michael Stark1,2 György Szarvas1 Iryna Gurevych1 Bernt Schiele1,2
1Department of Computer Science, TU Darmstadt 2MPI Informatics, Saarbrücken
2
Knowledge transfer for zero-shot object class recognition
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
Group classes by attributes[Lampert et al., CVPR `09]
Manual supervision:Object class-attribute associations
Group classes by attributes[Lampert et al., CVPR `09]
Manual supervision:Object class-attribute associations
Describing using attributes[Farhadi et al., CVPR `09 & `10]
Manual supervision:Attribute labels
Describing using attributes[Farhadi et al., CVPR `09 & `10]
Manual supervision:Attribute labels
• animal• four legged• mammal
white paw
Unseen class(no training images) Giant panda ?
Attributes for knowledge transfer
3
Knowledge transfer for zero-shot object class recognition
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
Describing using attributes[Farhadi et al., CVPR `09 & `10]
Manual supervision:Attribute labels
Describing using attributes[Farhadi et al., CVPR `09 & `10]
Manual supervision:Attribute labels
• animal• four legged• mammal
Group classes by attributes[Lampert et al., CVPR `09]
Manual supervision:Object class-attribute associations
Group classes by attributes[Lampert et al., CVPR `09]
Manual supervision:Object class-attribute associations
white paw
Unseen class(no training images) Giant panda ? Attributes for knowledge transfer Replace manual supervision
by semantic relatednessmined from language resources
Unsupervised Transfer
WordNetAttributes for knowledge transfer
4
Attribute-based model [Lampert et al., CVPR `09]
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
oceanoceanspotsspots ……
Known training classes
Attribute classifiers
Unseentest classes
Class-attribute associations
Class-attribute associations
[Lampert et al., CVPR `09]
Supervised:manual (human judges)
Attributeswhitewhite
5
Attribute-based model [Lampert et al., CVPR `09]
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
WordNet
semantic relatednessfrom language
[Lampert et al., CVPR `09]
Supervised:manual (human judges)
oceanoceanspotsspots ……
Known training classes
Attribute classifiers
Unseentest classes
Class-attribute associations
Class-attribute associations
whitewhite
6
Direct similarity-based model
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
WordNet
semantic relatednessfrom language
oceanoceanspotsspots ……
Known training classes
Attribute classifiers
Unseentest classes
Class-attribute associations
Class-attribute associations
whitewhite
7
Direct similarity-based model
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
WordNet
semantic relatednessfrom language
Known training classes
Unseentest classes
Class-attribute associations
Classifierper class
killer whalekiller
whaleDalmatian polar
bearpolar bear
8
Direct similarity-based model
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
WordNet
semantic relatednessfrom language
Unseentest classes
most similarclasses
Known training classes
Classifierper class
polar bearpolar bear
killer whalekiller
whaleDalmatian……
9
Models for visual knowledge transferSemantic relatedness measuresLanguage resources
WordNet Wikipedia WWW Image search
Respective state-of-the-art measuresEvaluationConclusion
Outline
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
10
WordNet Lin measure
[Budanitsky & Hirst, CL `06]Wikipedia Explicit Semantic Analysis
[Gabrilovich & MarkovitchI, IJCAI `07]
Word Wide Web Hitcount (Dice coeffient)
[Kilgarriff & Grefenstette, CL `03]
Image search Visually more relevant Hitcount (Dice coeffient)
Semantic Relatedness Measures
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
WordNet[Fellbaum, MIT press `98]
WordNet[Fellbaum, MIT press `98]
11
WordNet Lin measure
[Budanitsky & Hirst, CL `06]Wikipedia Explicit Semantic Analysis
[Gabrilovich & MarkovitchI, IJCAI `07]
Word Wide Web Hitcount (Dice coeffient)
[Kilgarriff & Grefenstette, CL `03]
Image search Visually more relevant Hitcount (Dice coeffient)
Semantic Relatedness Measures
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
WordNet[Fellbaum, MIT press `98]
WordNet[Fellbaum, MIT press `98]
12
WordNet Lin measure
[Budanitsky & Hirst, CL `06]Wikipedia Explicit Semantic Analysis
[Gabrilovich & MarkovitchI, IJCAI `07]
Word Wide Web Hitcount (Dice coeffient)
[Kilgarriff & Grefenstette, CL `03]
Image search Visually more relevant Hitcount (Dice coeffient)
Semantic Relatedness Measures
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
A farm is an area of lanthe training of horses.
A hoof is the tip of a toe
Rear hooves of a horse
Hoof
Farm
Tusks are long teeth, uElephants and narwhals
Tusk
Article horse elephantFarm 3 0Hoof 2 1Tusk 0 4
… … …
A farm is an area of lanthe training of horses.
A hoof is the tip of a toe
Rear hooves of a horse Most evem tped ungulat
Hoof
Farm
Tusks are long teeth, uElephants and narwhals
Tusk
13
WordNet Lin measure
[Budanitsky & Hirst, CL `06]Wikipedia Explicit Semantic Analysis
[Gabrilovich & MarkovitchI, IJCAI `07]
Word Wide Web Hitcount (Dice coeffient)
[Kilgarriff & Grefenstette, CL `03]
Image search Visually more relevant Hitcount (Dice coeffient)
Semantic Relatedness Measures
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
A farm is an area of lanthe training of horses.
A hoof is the tip of a toe
Rear hooves of a horse
Hoof
Farm
Tusks are long teeth, uElephants and narwhals
Tusk
Article horse elephantFarm 3 0Hoof 2 1Tusk 0 4
… … …
A farm is an area of lanthe training of horses.
A hoof is the tip of a toe
Rear hooves of a horse Most evem tped ungulat
Hoof
Farm
Tusks are long teeth, uElephants and narwhals
Tusk
cosine
14
WordNet Lin measure
[Budanitsky & Hirst, CL `06]Wikipedia Explicit Semantic Analysis
[Gabrilovich & MarkovitchI, IJCAI `07]
Word Wide Web Hitcount (Dice coeffient)
[Kilgarriff & Grefenstette, CL `03]
Image search Visually more relevant Hitcount (Dice coeffient)
Semantic Relatedness Measures
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
15
WordNet Lin measure
[Budanitsky & Hirst, CL `06]Wikipedia Explicit Semantic Analysis
[Gabrilovich & MarkovitchI, IJCAI `07]
Word Wide Web Hitcount (Dice coeffient)
[Kilgarriff & Grefenstette, CL `03]
Image search Visually more relevant Hitcount (Dice coeffient)
Semantic Relatedness Measures
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
We watched a horse race yesterday. [..] Tomorrow we go in the zoo to look at the baby elephant.
„the dance of the horse and elephant“
web search image search[http://www.flickr.com/photos/ lahierophant/2099973716/]
Incidental co-occurence
Terms refer to same entity (the image)
16
Models for visual knowledge transferSemantic relatedness measuresEvaluationAttributes Querying class-attribute associations Mining attributes
Direct similarityAttribute-based vs. direct similarity
Conclusion
Outline
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
17
Animals with attributes dataset [Lampert et al., CVPR `09]40 training, 10 test classes (disjoint)≈ 30.000 images totalDownsampled to 92 training images per classManual associations to 85 attributes
Image classificationSVM: Histogram intersection kernelArea under ROC curve (AUC) - chance level: 50%Mean over all 10 test classes
Experimental Setup
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
18
Performance of supervised approach
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
19
Querying: abbreviationagile
Manual supervision: detailed description“having a high degree of physical coordination”
Querying: abbreviationagile
Manual supervision: detailed description“having a high degree of physical coordination”
Performance of queried association Encouraging Below manual supervision
Image search Based on image related text
Wikipedia Robust resource
Yahoo Web Very noisy resource
WordNet Path length poor indicator of
class-attribute associations
Querying class-attribute association
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
20
Performance of queried association Encouraging Below manual supervision
Image search (Yahoo Img, Flickr) Based on image related text
Wikipedia Robust resource (definition texts)
Yahoo Web Very noisy resource
WordNet Path length poor indicator of
class-attribute associations
Querying class-attribute association
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
the dance of the horse and elephant
image search
21
Performance of queried association Encouraging Below manual supervision
Image search (Yahoo Img, Flickr) Based on image related text
Wikipedia Robust resource (definition text)
Yahoo Web Very noisy resource
WordNet Path length poor indicator of
class-attribute associations
Querying class-attribute association
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
Noise:While he watched a horse race
the leg of his chair broke.
Noise:While he watched a horse race
the leg of his chair broke.
22
Performance of queried association Encouraging Below manual supervision
Image search (Yahoo Img, Flickr) Based on image related text
Wikipedia Robust resource (definition text)
Yahoo Web Very noisy resource
WordNet Path length poor indicator of
class-attribute associations
Querying class-attribute association
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
23
Mining attributes
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
WordNet
semantic relatednessfrom language
Attributeterms oceanoceanspotsspots ……
Known training classes
Unseentest classes
Class-attribute associations
Class-attribute associations
whitewhite
24
Mining attributes
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
WordNet
semantic relatednessfrom language
Attributeterms ???? ??
Known training classes
Unseentest classes
Class-attribute associations
Class-attribute associations
??
25
Part attributesLeg of a dogAttribute classifiers
Mining attributes
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
WordNet
semantic relatednessfrom language
Known training classes
Unseentest classes
Class-attribute associations
Class-attribute associations
flipperleg paw WordNet
26
Additional measure:Holonym patterns Only part attributesHit Counts of Patterns
[Berland & Charniak, ACL 1999] “cow’s leg” “leg of a cow” Dice coefficient
Mining attributes
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
While he watched a horse race the leg of his chair broke.
Leg of the horse
web search holonym patterns
Incidental co-occurence
One term likely part of other term
27
Best: Yahoo Holonyms Close to manual attributes Tailored towards part attributes
Mining attributes
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
28
Best: Yahoo Holonyms Close to manual attributes Tailored towards part attributes
Performance drop Reduced diversity Only part attributes
Specialized terms E.g. pilus (=hair) Coverage problem:
Image search, Wikipedia
Mining attributes
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
29
Models for visual knowledge transferSemantic relatedness measuresEvaluationAttributes Querying class-attribute associations Mining attributes
Direct similarityAttribute-based vs. direct similarity
Conclusion
Outline
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
30
Direct similarity-based model
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
WordNet
semantic relatednessfrom language
Unseentest classes
most similarclasses
Known training classes
Classifierper class
polar bearpolar bear
killer whalekiller
whaleDalmatian
31
Nearly all very good On par with manual supervision
attribute model (black) Clearly better than any
mined attribute-associations result
Why? Five most related classes Ranking of semantic
relatedness reliable Similar between methods
Direct Similarity
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
32
0 10.000 20.000
65
70
75
80
mea
n AU
C (in
%)
Number of additional training class images in test set
attributes: manually definedattributes: mined associationsattributes: mined attributesdirect similarity
Attributes vs. direct similarity
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
Extending the test setAdd images From known classes As negatives
More realistic setting
ResultsDirect similarity
drop in performance(orange curve)
Attribute modelsgeneralize well
0 10.000 20.000
65
70
75
80
mea
n AU
C (in
%)
Number of additional training class images in test set
attributes: manually definedattributes: queried associationsattributes: mined attributesdirect similarity
0 10.000 20.000
65
70
75
80
mea
n AU
C (in
%)
Number of additional training class images in test set
attributes: manually definedattributes: queried associationsattributes: mined attributesdirect similarity
0 10.000 20.000
65
70
75
80
mea
n AU
C (in
%)
Number of additional training class images in test set
attributes: manually definedattributes: queried associationsattributes: mined attributesdirect similarity
0 10,000 20,000
65
70
75
80
mea
n AU
C (in
%)
Number of additional training class images in test set
attributes: manually definedattributes: mined associationsattributes: mined attributesdirect similarity
33
Models for visual knowledge transferSemantic relatedness measuresEvaluationConclusion
Outline
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
34
Supervision replaced with semantic relatedness Direct similarity better than attributes on par with supervised approach
Attributes: generalizes better
Semantic relatedness measures Overall best Yahoo image with hit count
Holonym patterns for web search Improvement Limited to part attributes
Conclusion
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
35
Supervision replaced with semantic relatedness Direct similarity better than attributes on par with supervised approach
Attributes: generalize better
Semantic relatedness measures Overall best Yahoo image with hit count
Holonym patterns for web search Improvement Limited to part attributes
Conclusion
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
0 10.000 20.000
65
70
75
80
mea
n AU
C (in
%)
Number of additional training class images in test set
attributes: manually definedattributes: mined associationsattributes: mined attributesdirect similarity
0 10.000 20.000
65
70
75
80
mea
n AU
C (in
%)
Number of additional training class images in test set
attributes: manually definedattributes: queried associationsattributes: mined attributesdirect similarity
0 10.000 20.000
65
70
75
80
mea
n AU
C (in
%)
Number of additional training class images in test set
attributes: manually definedattributes: queried associationsattributes: mined attributesdirect similarity
0 10.000 20.000
65
70
75
80
mea
n AU
C (in
%)
Number of additional training class images in test set
attributes: manually definedattributes: queried associationsattributes: mined attributesdirect similarity
0 10.000 20.000
65
70
75
80
mea
n AU
C (in
%)
Number of additional training class images in test set
attributes: manually definedattributes: queried associationsattributes: mined attributesdirect similarity
36
Supervision replaced with semantic relatedness Direct similarity better than attributes on par with supervised approach
Attributes: generalize better
Semantic relatedness measures Overall best Yahoo image with hit count
Holonym patterns for web search Improvement Limited to part attributes
WordNet poor for object-attributes associations
Conclusion
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
patterns:dog’s legleg of the dogs
patterns:dog’s legleg of the dogs
0 10.000 20.000
65
70
75
80
mea
n AU
C (in
%)
Number of additional training class images in test set
attributes: manually definedattributes: mined associationsattributes: mined attributesdirect similarity
0 10.000 20.000
65
70
75
80
mea
n AU
C (in
%)
Number of additional training class images in test set
attributes: manually definedattributes: queried associationsattributes: mined attributesdirect similarity
0 10.000 20.000
65
70
75
80
mea
n AU
C (in
%)
Number of additional training class images in test set
attributes: manually definedattributes: queried associationsattributes: mined attributesdirect similarity
0 10.000 20.000
65
70
75
80
mea
n AU
C (in
%)
Number of additional training class images in test set
attributes: manually definedattributes: queried associationsattributes: mined attributesdirect similarity
0 10.000 20.000
65
70
75
80
mea
n AU
C (in
%)
Number of additional training class images in test set
attributes: manually definedattributes: queried associationsattributes: mined attributesdirect similarity
37
Further supervision for closing the semantic gap?See us at our poster (A2, Atrium)!
CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |
Knowledge Transfer
Thank you!
Software? www.mis.tu-darmstadt.de/nlp4vision