e-culture: challenging use cases for the semantic webguus/talks/05-eswc-workshop.pdf · the turning...
TRANSCRIPT
E-Culture: Challenging Use Cases for the Semantic Web
Guus SchreiberFree University Amsterdam
2
Project contextMany examples are based on work in the Dutch BSIK project Multimedian http://multimedian.nl
3
Purpose
Analyze a number of use cases from e-culture domain– Multimedia plays key role
Required technology– Typically combination of technologies
Relation to state of the art
4
Use case: Asian chairsUser has found an image of an Asian chair
Annotation:ex:image vra:stylePeriod aat:Guangxu .
How can we find images of Asian chairs from the same historical period?
5
AAT info on Guangxu
6
Importance of time and space information
Many queries require time/space knowledge, either absolute or abstractedFor the chair image we can establish– Country = China (link Chinese => China)– Period = 1644-1911 (from Qing description)
Technology requirements:– Thesuari relating time/space concepts– NLP for unstructured descriptions– Time/space reasoning techniques
7
8
9
Sample place information in TGN
<tgn:AdministrativePlace rdf:about="&tgn;1000111"tgn:standardLatitude="35"tgn:standardLongitude="105“>
<vp:parentPreferred rdf:resource="&tgn;1000004"/>……..
</tgn:AdministrativePlace>
10
Issues when searching for “nearby” Asian chairs
Close in space:– Other country in (East) Asia– Latitude/longitude
Close in time:– Links between style periods– Match time periods (and
handle incomplete information)
11
12
Use case: painting style
Find paintings of a similar style
MATISSE, HenriLe bonheur de vivre (The Joy of Life)1905-1906Oil on canvas, 69 1/8 x 94 7/8 in. (175 x 241 cm)Barnes Foundation, Merion, PA
13
How can we find this other Fauve painting?
DERAIN, AndreThe Turning Road, L'Estaque, 1906Oil on canvas, 51 x 76 3/4 in. (129.5 x 195 cm)Museum of Fine Arts, Houston, Texas
14
IssuesParse annotation to find matches with thesauri terms– E.g. match artists to ULAN individuals
Artists-style links– AAT contains styles; ULAN contains artists, but there
is no link• Learn link from corpora• Derive it from other annotations
– Domain-specific rules/reasoning needed • see example in SWRL doc• Painters may have painted in multiple styles
15
16
17
Issues w.r.t. thesauri
Public availability!RDF/OWL representationLearning/specifying term/concept mapping– owl:equivalentClass, owl:sameAs, rdf:type, rdfs:subClassOf
– Domain-specific linksManaging the evolution of the thesauri and the mappings
18
Use case: find images with the same subject
Find another painting which portrays dancing
19
Issues
Same subjects can be visually very differentSubject is often missing from the annotationMismatch: user often search for subjects of images
20
Conceptual subject descriptions85% of the user queries:
General Descriptions of generally known items. Only general, everyday knowledge is necessary. Descriptions are at the level of the Natural categories of E. Rosch (1973), or more general. E.g An ape eating a banana.
Specific Descriptions of objects or scenes that can be identified and named. Specific domain knowledge is necessary to recognize the objects or scenes. E.g. The old male gorilla Kumba, born in Cameroon and now living in Artis, Amsterdam
Abstract Descriptions for which interpretative knowledge is used. This category is subjective. E.g An animal threatened with extinction.
21
Example concepts in image
Specific– Fall of the Berlin Wall
General– People walking at night
Abstract– Fall of the Iron Curtain
22
Use of conceptual categories by people searching for images
Conceptual level: 83%
0%
20%
40%
60%
80%
100%
event time place relation scene object
Characteristics
Nub
er o
f ele
men
ts in
% o
f co
ncep
tual
ele
men
ts
AbstractSpecificGeneral
23
Thesauri for scenes: Iconclass
24
25
26
Annotation of image content
Template for subject descriptionAgent Action Object Recipient
Guidelines for manual annotation– Annotate as specific as possible
Default reasoningCBIR support:– Object identification– Spatial relations
27
28
29
Some forms of image content are well suited to image analysis
Collection of clothesAbstract painting
30
The semantic gapThe distance between Content-Based Image Retrieval and semantics:– Smeulders, Worring, Santini, Gupta, Jain. Content-
based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12), December 2000.
Direct links between visual features and semantic concepts become more difficult when the domain is broader / more general
31
Example semantic bridge:microscopic cell images
mpeg7 : StillRegion(region) ^mpeg7x : Dense(region) ^mpeg7 : DominantColor(region, col) ^swrlb : lessThan(col, 100)
=> mpeg7 : Depicts(region, mesh : MatureGranule)
32
Segmentation often requires user interaction
33
Automatic detection of concepts can be difficult even in “easy” cases
What is the color of this ape?
34
Image analysis useful for collection navigation
35
Bridging the semantic gap:CBIR and ontologies
Visual WordNet (GE paper)– Adding knowledge about visual characteristics
to WordNet: mobility, color, …– Build detectors for the visual features– Use visual data to prune the tree of categories
when analyzing a visual object
36
Sample visual features and their mapping to WordNet
37
Experiment: pruning the search for “conveyance” concepts
6 concepts foundIncluding taxi cab
12 concepts foundIncluding passenger train and commuter train
Three visual features: material, motion, environment Assumption is that these work perfectly
38
Bridging the semantic gap:concept detectors
Snoek et al., TRECVID2004– 185 hours of news video
32 detectors for concepts in news video– Through machine learning
Similarity detectors based on keywordsand visual analysisQuery interface in which these functions can be combined
39
“Concepts” for which visual detectors were built
40
41
TRECVID 2004 results
42
Main observation
A combination of many different techniques is needed to be able to cope with the complexity of multimedia semantics– NLP, segmentation, CBIR, visual feature
detectors, visual ontologies, publicly available thesauri, thesauri mappings, dedicated reasoning techniques (time, space, default), personalizaion, presentaion generaion
Key role for user studies