with contributions by many west membersstaab/presentations/...kernel support vector machines hong et...

Web Science & TechnologiesUniversity of Koblenz ▪ Landau, Germany

Multimedia WebSteffen Staab

With contributions by many WeST members

Steffen [email protected]

CV Integration: Multimedia Web2 of 90

WeST

Semantic WebDr. Janik

Web RetrievalDr. Sizov

Interactive WebDr. Scherp

Multimedia WebDr. Grzegorzek

Software WebF. Silva Parreiras

Prof. Dr. Staab Prof. Dr. Sure

EU WeGovEU X-Media

HP Synth DocsEU WeKnowItDFG Multipla EU Most

BMBF CollabCloudEU NeOn

EU Tagora

Wer sind wir?

GESISProf. Sure

EU WeGovEU ASG

EU aceMediaEU kspace

EU Net2



WeST

Groups

Caption

Multimedia in context

Time

Low-level

UserProfile

Favs

Comms

Geo Social

network

Tags



WeST

Multimedia Web

Daten

Interaktion Personen

Wissen

Netzwerke

Retrieval

Metadaten

Signalverarbeitung

Verstehen

Präsentieren

Multimediadatenbanken

Web Science

Information Retrieval

Pattern Recognition

Interaktive Multimediasysteme



WeST

Agenda

Multi-modal analysis of social media

Large Scale Tag Recommendation

Understanding the Semantics of Images

Spatial and OntologicalReasoning For Image Understanding


Large Scale Tag Recommendation

Rabeeh Abbasi



WeST

Recommending Tags

Tag RecommenderTag Recommender

Big BenSt. Stephen's TowerClock TowerPalace of WestminsterHouses of ParliamentWestminsterLondonUnited Kingdom

Tags

InputInput OutputOutput

Geographical Coordinates

51°30'03"N0°7'24"W

Low-level Features

Which image features to use?



WeST

System Overview

Tag RecommenderTag Recommender

51°30'03"N0°7'24"W

Big BenSt. Stephen's TowerClock TowerPalace of WestminsterHouses of ParliamentWestminsterLondonUnited Kingdom

TrainingData

Location 1

Location N

Location 1Clusters

Location NClusters

Model

Features

Features

Image Features

Tags



WeST

• Tags• Flickr tags assigned by the users

• Geographical Coordinates• Latitude and Longitude

• Low-level Image Features

Features in Social Media



WeST1 2 3 4 5 10 15 20

0

0.02

0.04

0.06

0.08

0.1

0.0751

0.0859

0.09140.0936

0.0892

0.0750

0.0645

0.0581

0.0272

0.0376

0.04350.0464 0.0474 0.0475

0.04480.0417

0.0244

0.0323 0.0338 0.0346 0.03490.0311

0.02810.0256

0.0183

0.02580.0300

0.0337 0.0357 0.0373 0.0358 0.0341

Geo EHD Tags Random

Number of tags recommended

F-M

easu

re

F-Measure

Considered ~400,000 images from 58 national capitals75% training / 25% test (gold standard) data

Geo CoordinatesLow-level FeaturesTagsBaseline (Random)



WeST

Conclusions

New images can be automatically tagged provided their geographical coordinatesGeographical coordinates give best tag recommendations

Require less computational power (just 2 dimensions)


Multi-modal Analysis of Social Media

Dr. Dr. Sergej Sizov



WeST

Groups

Caption

Multi-modal Analysis of Social Media

Time

Low-level

UserProfile

Favs

Comms

Geo Social

network

Tags



WeST

1. Select characteristic aspects of social media: tags, timestamps, coordinates, low-level features, etc.)

2. Construct probabilistic modelsusing joint distributions of considered aspects(latent semantic analysis)

3. Compute topic-basedprobabilistic feature vectorsfor resources, tags, and users in our model

4. Construct applications that exploit these feature vectorsfor content categorization, filtering, or search

What we do with all this..



WeST

Case 1: recommending cool photos to users

Global Flickr model Personal User-specific model

User postings are NOT characteristic for user interests!They are looking for something quite different ..

.. published at ACM Hypertext 2009

Can we predict and recommend favorites?

Lesson1



WeST

Case 2: detecting events

„Unusual“ topics localized in time AND space indicate events- anomalies are interesting and mean something special ..

.. to appear in DB Spektrum, Special Issue on Social Media

Selected latent topics in the Flickr! Barcelona dataset 2005-2008These topics are only active in small time/space segments (1 week - 1x1 km)

microsoft 0.139teched 0.136autumn 0.132student 0.132partners 0.132msp 0.132nov 0.125water 0.031bycicle 0.017food 0.010

graffiti 0.199streetart 0.199art 0.199street 0.199mtn 0.012happy 0.012blameless 0.011montana 0.009parr 0.007spraypaint 0.003

girls 0.110show 0.108sexy 0.108ficeb 0.108erotic 0.107model 0.107festival 0.107ficeb07 0.107tattoo 0.105fishnet 0.019

3gsm 0.408fira 0.335fair 0.073trade 0.058montjuic 0.051fountains 0.037amazing 0.029show 0.027night 0.012water 0.011

deportivo 0.077sports 0.076camp 0.063soccer 0.055lacoru 0.055futbol 0.054liga 0.052deportes 0.041barca 0.038depor 0.036campnou 0.036

Lesson2



WeST

Case 3: exploiting location awareness

Geo coordinates can supplement text featuresin search and retrieval apps

.. published at ACM WSDM 2010

Can we exploit tag-location correlations (i.e. text features and GPS coordinates)for better content organization? Example: Flickr London dataset

classifying photos clustering photos

automatic tag recommendation

Lesson3


Understanding the Semantics of Images

Dr. Marcin Grzegorzek



WeST

The inability to access accurate and desired content can be as limiting as the lack of content itself!

Textual information : Human defined and precise in meaningAV content : Hidden component of human creative reasoning The task:

– Automatic annotation and retrieval of AV content using semantic structures natural to humans (e.g. words or sentences);

Challenges:– Subjective interpretation of content by different users

under different conditions; – Discover links between semantic based query and low-

level metadata.



WeST

Motivation

Human user :Semantic similarity

Machine:Low-level similarity

The approach:• Merging different low-level features;• Dealing with the subjective interpretation of content;• Modeling semantic objects as mosaics made of irregular structures obtained

through low-level and spatial clustering of building blocks.

The end goal: Incorporating the obtained knowledge into a learning approach and teaching the machine to “reason” as a human.



WeST

Descriptive Learning Models

Descriptive learning modelsrelevant class modelled with….

Single Gaussian distribution

Heuristic

Non-heuristicThe MARS system (Rui et al. 1998): re-weighting techniques and query point movement;

The PicHunter system (Cox et al. 1998), RF in a form of relative judgment and an extension of k-d trees.

Ishikawa et al. 1998: An optimization problem and considering correlation among features .

More complex distribution

Qian et al. 2002 : Gaussian Mixture Models, both relevant and irrelevant examples as well as unlabelled data

Parametric models

Non-parametric models

Non-parametric distribution for relevant samples

Parzen window density (Meilhac and Nastar 1999): RF problem as difference of densities for relevant and irrelevant samples; Incremental approach with use of user’s feedback for increasing precision of model estimation.



WeST

Discriminative Learning Models

Discriminative learning modelsestimating the boundary between classes…

Non-linearity of visual space= Discriminative approaches + Kernel based algorithms;

Discriminative Analysis

Linear DA approach tries to cluster all irrelevant images into one class, whereas negative examples belong to many different classes….

Wu et al. 2000:Transformation maximizes the ratio between inter-class and intra-class scattering.Disadvantage: Each negative example is treated as from a different class

Linear DA

Multiple DA

Biased DA

Zhou & Huang 2001:

Positive examples are clustered in one class while negative examples can come from an uncertain number of classes; Disadvantage: Sensitivity to imbalance between classes; considerable amount of training required.

Kernel support vector machines

Hong et al. 2000 & Guo et al. 2002 : The input data is mapped into higher-dimensional feature space using a non-linear transform- kernel; In the newly defined space a boundary between classes can be easily found…

…



WeST

System Overview

Multi-FeatureSpace

Automatic Feature

Extraction

Sub-System I

Learning Approach (SVM)‏

Adaptive Convolution Kernel

User

RetrievedImages

Sub-System III

RepresentativeStructured Multi-

Feature Sspace

Set of Blocks

Sub-System II



WeST

Object Modelling

Semantic objects can be regarded as mosaics of small building blocks

STEP1: Elementary overlapping structures covering the whole picture;

STEP2: If a block is a member of two or more neighbouring structures then it is removed from all structures but the closest one (low-level similarity)Breakdown into non-regular, non-overlapping structures

The 3x3 neighbourhood used to build initial regular structures and its breakdown into non-regular, non-overlapping structures

ijS ijS%

(i,j) (i,j+1)

(i+1,j) (i+1,j+1)

(i-1,j-2) (i-1, j-1) (i-1,j)

(i,j-2) (i,j-1)

(i-1,j+1) (i-1,j+2)

(i,j+2)

(i+1,j+2)

(i-1,j-1) (i-1,j) (i-1,j+1)

(i,j-1) (i,j) (i,j+1)

(i+1,j-1) (i+1,j) (i+1,j+1)

STEP3: Key-representative structures within each image are extracted using k-medoids clustering method and low-level distance as well as the spatial proximity of generated structures;



WeST

Databases for Evaluation

Two different databases are used for evaluationA subset from the Corel stock consisting of seven concepts, these arebuildings, clouds, cars, elephants, grass, lions and tigers:

25 distinct classes from the COIL database:


Spatial and OntologicalReasoning For Image Understanding

Carsten Saathoff



WeST

Motivation

Local features often not sufficient for classificationExploit explicitly defined spatial knowledge to improve labelling

e.g. Sky not allowed left or right of SeaAllow for efficient training of classifiers and spatial knowledge

Good labelling performance with few training examples

Sky

Sea

Sea

Sand

SeaSeaSky ↯



WeST

Analysis Framework

HypothesesGeneration

Spatial Relations Extraction

SpatialReasoning

Training

Analysis

(x,x,x,x)(x,x,x,x)(x,x,x,x)(x,x,x,x)

Low LevelClassifiers

SkySea above

Sky

Sand

SeaSea

Trained SpatialBackground Knowledge

Training Examples



WeST

Exploiting Spatial Context

Create optimization problem fromRegionsSpatial RelationsHypotheses setsSpatial background knowledge

ApproachesFuzzy Constraint Satisfaction

• WIAMIS08Binary Integer Programming

• SAMT2008

Sand, 0.8Sea, 0.7Person,0.5…

Hypotheses

Background Knowledgeabove-of (Sky, Sea) -> 1.0

(Sea, Sand) -> 1.0(Sea, Sky) -> 0.0…

above above

above above

left-of



WeST

Experimental Setup

Measure improvement achieved withFCSPLinear Programming

over low-level classification with different training set sizesData set with 923 natural and urban imagesSet of 10 concepts

building,foliage,mountain,person,road,sailing-boat,sand,sea,sky,snow5690 labelled regions568 labelled „unknown“ -> ignored

Ground truth defined on automatic segmentationRegions labelled with dominant conceptGround truth created for this work



WeST

Overview of dataset

Available at: http://mklab.iti.gr/project/scef



WeST

Classification Rate over Training Set Size

50 100 150 200 250 300 350 4000.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

Low-level classi-ficationFCSPBIP



WeST

F-Measure per Concept

Training Set with 300 images (best performing one for BIP)

snow road mountain foliage person sand sky sailing-boat sea building average0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Low-levelFCSPBIP



WeST

Ontological Background Knowledge

Question: Can we integrate general knowledge into theoptimization problem?Examples:

Beach Images contain some Sky region and some SearegionAn outdoor image taken on Mallorca in summer does notcontain any Snow

ToolsDescription LogicsBinary Integer Programs



WeST

Exploiting general background knowledge

Generic image interpretationCreate optimization problem from

Hypotheses about imageRelations between individuals describing the imageOntological background knowledge

Approach based on extension of BIP reasoner

Sand, 0.8Sea, 0.7Person,0.5…

Background Knowledge

above above

above above

left-of

image

Beach, 0.8Mountain, 0.7Indoor,0.5…

depicts

region

Beach = depicts some Seaand depicts some Sand

Sea = above only (Sand orSea or Boat)



WeST

Conclusion

Still more work:Dr. Ansgar Scherp

• Metadata standards for SMIL, Flash, etc.• Modeling of Events• User interaction with multimedia

Thomas Franz: Semantic relationships between information objectsSimon Schenk:Semantic backend technology….


Semaplorer



WeST

Many sources – little context

Links Location

Persons

Knowledge Tags

low- to midlevel features

xxxxxxxxx



WeST

Choosing between Koblenz – and Koblenz



WeST

Contextual Information



WeST

Tag-based refinement



WeST

A tag view of „Koblenz“ & „Castle“



WeST

Semantic Identity – Festung Ehrenbreitstein



WeST

Persons – Celebrities, FOAFers & Flickr Users



WeST

Some Shades of Multimedia Meaning

Links Location

Persons

Knowledge Tags


xxxxxxxxx

GeoNames



WeST

Conclusion

Multimedia Web isSignal Processing

Encoding, wavelets, bitmaps,…Machine Learning

Clustering, classification, Bayes, SVM, LDA,…Social Context

Recommender systems, favorites, Web 2.0 applications…Background Knowledge

Ontologies, logics, optimization, …Interactive Systems

Systems, user studies, graphics, …Fun!


Thank You!



WeST

Agenda

Semaplorer: Faceted Browsing of Semantic Multimedia Data

Linked Open Data by Collective Intelligence

Semaplorer Architecture

Linking High-Level and Low-Level Data

Representational Paradigm• COMM – Core Ontology for MultiMedia• F – Event Ontology

Computational Paradigm

Links Location

Persons

Knowledge Tags


xxxxxxxxx



WeST

Collective Intelligence

Collective datasetsHosted public datasetsGated datasets

• Social networks,…

Wikipedia styleActually includes

• Discussions• Editor hierarchies• Policies

Pagerank stylehighly effectiveno coordinationno control (modulo spamming)

Gene ontologyDBPedia, Public census dataFacebook, LinkedIn

WikiversityFAQs

Yahoo Answers, Lycos IQ

TaggingFlickr, Delicious, …geotagging

Different Flavors Linked Open Data



WeST

Linked Open Data: Example Instances



WeST

Linked Open Data: Instance Containers



WeST

Linked Open Data: Classes



WeST

Agenda







Links Location

Persons

Knowledge Tags


xxxxxxxxx



WeST

Swoogle

RDFSrulesgeo...

Billion triples challenge: Use Linked Open Data

Common approach: Import dump to new data silo

Semantic Web?

Geoquerying

GeoNames

WordNet

GeoNames

flexible

scaleable

webby

extensible

RDFS Rules

inflexiblemonolithic

notscaleable

PlaceOfBirthbirthplace

birthplace

WordNet Swoogle

fulltext

12 months in 2005/06700M triples

+ ++

+ >1Gt

Since mid Nov: Flickr API

Since Dec, Amazonpart shut down



WeST

A federated RDF repository

GeoNames

RDFS Rules

PlaceOfBirthbirthplace

birthplace

WordNet Swoogle

fulltext

12 months in 2005/06700M triples

+ ++

+ >1Gt

Since mid Nov: Flickr API



WeST

Architecture of Federated Infrastructure

Semantic Annotation Tool KATExtended by generic and application-specific pluginsProviding SemaPlorer‘s UIPlugin: Download image content from flickrPlugin: Map component using OpenStreetMap



WeST


Views:?person birthplace ?city=>?person birthplace ?city UNION?person placeOfBirth ?city

Query Splitting:?geoSight ?geo:lat ?lat; geo:long ?long.?geoSight owl:sameAs ?dbpSight.<Category:VisitorAttraction> skos:broader ?dbpSight.=>1) ?geoSight ?geo:lat ?lat; geo:long ?long.

?geoSight owl:sameAs ?dbpSight.DISTRIBUTED JOIN2) <Category:VisitorAttraction> skos:broader ?dbpSight.

Locating endpoints:graphname endpoint



WeST


Endpoints:•Autonomous RDF repositories•Connected via SPARQL•Dynamic (added by reconfiguring SourceFinder)•25 repositories, > 400GB including fulltext indices

Inferencing and Querying:Can vary from repository to repository

Here:•SPARQL based Views•RDFS•transitivity rules for SKOS•Geo range queries in SPARQL•fulltext search in SPARQL via Lucene



WeST

Architecture of Federated InfrastructureControl EC2 Instances:

Transparent Reconfiguration:

Administration Component updates Souce Finder Configuration, no need for the application to know.

Repositories:

10 running at Koblenz University, Germany

15 running at Amazon EC2, USA



WeST

SemaplorerSemaplorer

� Scaleable � Flexible� Federated

Semantic Web infrastructure using� RDFS� Rules� Views� Geoqueries� Fulltext Search� Based on� Cloud Computing and� NetworkedGraphs

http://btc.isweb.uni-koblenz.de



WeST

Agenda





Representational Paradigm• COMM – Core Ontology for MultiMedia


Links Location

Persons

Knowledge Tags


xxxxxxxxx



WeST

Yalta Conference

Winston ChurchillRecognizer

Franklin D. RooseveltRecognizer

Josef StalinRecognizer

Linking Low-Level Data, Too: Multiple Tools



WeST

http://en.wikipedia.org/wiki/Yalta_Conference

World War II

Yalta ...

...History Ontology

Creating a Multimedia Presentation

SR1 SR2 SR3




ChurchillRooseveltStalin

WeST

Issues<Mpeg7><Description xsi:type="ContentEntityType"><MultimediaContent xsi:type=„ImageType"><Image><SpatialDecomposition>

<StillRegion id=„SR1“><TextAnnotation><KeywordAnnotation xml:lang="en"><Keyword>Churchill</Keyword>

</KeywordAnnotation></TextAnnotation>

</StillRegion>

<StillRegion id=„SR2“><Semantic><Label><Name>Roosevelt</Name>

<Label></Semantic>

</StillRegion>

<StillRegion id=„SR3“><Semantic><Definition> <StructuredAnnotation><WhatObject><Name xml:lang="en">Stalin</Name>

</WhatObject></StructuredAnnotation>

</Definition></Semantic>

</StillRegion>...

How do you formulate a query to get images

showing Churchill et al.?

First Shot (Xpath)://StillRegion[.//Keyword=“Churchill” or

.//Keyword=”Roosevelt” or

.//Keyword=”Stalin”]


Franklin RooseveltRecognizer



WeST

What is the Problem with MPEG-7?

Annotations are not interoperable!Ambiguities due to complementary description tools

Multiple ways to model semantically identical descriptions

Several alternatives for placing description tools inside an annotation

Complex queries needed to cover all alternatives!

Incompatible with (semantic) web technologies!You cannot link to the outside world!



WeST

Capabilities and Maturity Levels

Integration Automation

Former Situation

Current Situation

Future / Desired Situation

no standard, no vocabularymanual 1:1 agreement on format and semanticstight coupling of data and applications

standard vocabularymanual 1:1 agreement onmpeg-7 vocabularytight coupling of data and applications

standard vocabularypre-defined meaningad-hoc coupling of data and applications

Formerly

MPEG-7

COMM



WeST

MPEG-7

COMM

Requirements on a high quality

OntologyChallenge

BuildingBlock

Legend



WeST

TextDescriptor

MusicManager

CompoundDocument

Requirements for COMM

ReusabilityDesign a core ontology for any multimediarelated applicationMPEG-7-ComplianceSupport most important description toolsExtensibilityEnable inclusion of further

• description tools(even those that are not part of MPEG-7!)‏

• media typesSeparation of ConcernsClear separation of domain knowledge andknowledge about structureModularityEnable customization of multimedia ontologyHigh degree of axiomatization Ensure interoperability throughmachine accessible semantics

ChurchillRecognizer


FaceDetector

PhotoManager

AuthoringTool

SemanticAnnotation

decomposition visual descriptorsaudio descriptors ...

<Mpeg7>...

</Mpeg7>

<Mpeg7>...

</Mpeg7>



WeST

MPEG-7

COMM

Requirements on a high quality

MM Ontology

Quality of Ontologies

Challenge

BuildingBlock

Legend



WeST

How to Design a High Quality Multimedia Ontology?

Approach from [Oberle, 2005], [Oberle et al., 2006]:Use a well designed foundational ontology as a modelling basis to avoid shortcomings

Foundational ontologies provideFormal precisionDomain independenceBroad scope

Building upon foundational ontologiesprevents easy inclusion of modeling artefactsreduces conceptual ambiguityinherit rich axiomatization



WeST

Methodology

MPEG-7

COMM

Requirements: High Quality MM Ontology


Quality Measures for Ontologies

Reference Ontology

MPEG-7Compliance

Challenge

BuildingBlock

Legend



WeST

Methodology for Design Pattern Definition

Identification of most important MPEG-7 functionalities[Arndt et al., 2007]:

Decomposition of multimedia content into segmentsAnnotation of segments with meta data (e.g. visual descriptor, media information, creation & production, …)‏General: • Identify repetitive structures and

describe them at an abstract level• Describe digital data by digital data

at an arbitrary level of granularity

• Additional patterns are needed for:Complex data types of MPEG-7Semantic annotation by using domain ontologies

Interface between reusable multimedia core and domain specific knowledge



WeST

DOLCE Design Patterns: OIO and D&S

DOLCE is a library of foundational ontologies that provides 2 design patterns (extensions) that are especially important for MPEG-7:

Ontology of Information objects (OIO): Formalization of information exchange

Relevance for multimedia ontology:• MPEG-7 describes digital data (multimedia information objects) with

digital data (annotation) ‏• Digital data entities are information object

Descriptions & Situations (D&S): Formalization of ContextRelevance for multimedia ontology:

• Meaning of digital data depends on context • Digital data entities are connected through computational situations

(e.g. input and output data of an algorithm)• Algorithms are descriptions• Annotations and decompositions are situations that satisfy the rules

of an algorithm / method



WeST

Descriptions & Situations (D&S)‏

Distinction between:DOLCE ground entities (regions, endurants, perdurants) ‏Descriptive entities (parameters, roles, courses) ‏

DescriptionsFormalize contextDefine descriptive concepts

SituationsAre explained by descriptionsAre settings for ground entities



WeST

Putting it Together: Decomposition Pattern

Simply see the example on the next slide….



WeST

Decomposition by Example

Image1 playsRole SegmInput

Segment2 playsRole SegmOutp



Segment3 playsRole SegmOutpSegment1 playsRole SegmInput

Via its role in a computational task the different parts may be arbitrarily nested and related to different computing algorithms

Querying for all subparts takes place along a well-defined pattern



WeST

Methodology

MPEG-7

COMM

Requirements: High Quality MM Ontology


Quality Measures for Ontologies

Reference Ontologie

Identification of repetitive structuresMPEG-7

Compliance

Pattern definition through

Specialization

Challenge

BuildingBlock

Legend

Repr. of Context

Repr. of Information



WeST

Modular Architecture

Multimedia ontology consists ofCore module that contains the design patternsModules that specialize the core module for different media typesModules that contain media independent MPEG-7 description tools such as media information or creation & productionData type module that formalizes MPEG-7 data types e.g. matrices, vectors, unsigned-int-5, float-vector, probability-vector, …

DOLCE

Descriptions & Situation

Ontology of Information

Objects

Core

Visual Audio

Datatype

Media

Text / LingInfo

Domain Ontolog

Connected by SemanticAnnotation Pattern

Localization

Multimedia Knowledge (COMM)

Fundamental Knowledge about the World

Knowledge about a specific Domain



WeST

Does the Multimedia Ontology fulfil the Requirements?Reusability

Easy to queryMPEG-7-Compliance

Design patterns enable therepresentation of description tools

ExtensibilityDesign patterns are media independent possibility to include

• further media types• arbitrary descriptors

Extensions of multimedia ontologywill not affect legacy annotations dueto DOLCE+D&S+OIO

Separation of ConcernsClear separation between domainspecific and multimedia relatedknowledgeLink to Linked Open Data possible!

ModularityModular architecture allows customization

High degree of axiomatizationDesign patterns come with genericaxiomatization that is refined in derivedontology modules

ChurchillRecognizer


PhotoManager

COMM

One such extension has already beendone for Text Annotation.

Another one for compund documentannotation is currently developed!

Content & Media Annotation PatternSemantic Annotation Pattern

See slide before this slide!

OWL-DL version available for download.



WeST

http://en.wikipedia.org/wiki/Yalta_Conference

World War II

Yalta ...

...History Ontology

“Creating a Multimedia Presentation” Revisited

SR1 SR2 SR3




ChurchillRooseveltStalin

Sparql: select ?image where {?image plays AnnotatedDataRole. ?x plays SemanticLabelRole.?x rdf:type pol:President }

PhotoManager

AuthoringTool



WeST

Agenda







Links Location

Persons

Knowledge Tags


xxxxxxxxx



WeST

Computational Paradigm for Understanding High-level Semantics

Good to have: inside/outside, water,…EXIF!No „one approach fits all“Interesting directions:

Joining logical and probabilistic/fuzzy reasoning, e.g.– S. Dasiopoulou et al., SAMT 2008; Ralf Möller et al.

Joining collective intelligence and implicit high level semantics• Flickr, YouTube, etc.: An unprecedented set of training data!

– Several approaches for joining query log analysis / tags with low-level analysis

• Use semantic data that was not built for your task!

Just my two cents of speculation!



WeST

Conclusion: Recipe

Link existing data sources

Join existing computationalresources

Take Semantic Web as commondenominator

Invest a couple of person months

There you are!

Canonical representationCOMM – Multimediahttp://comm.semanticweb.org

F – EventsX-COSIMO – CommunicationCOS – Software



WeST

SSMS-2009

Advertisement: SSMS-2009Summer school on Social Media and Semantics

Primary target audience: PhD students & othersKoblenz, last week in August 2009

Thank you for your attention!http://isweb.uni-koblenz.de

http://btc.isweb.uni-koblenz.de



WeST

Get COMM

COMM is available online: http://comm.semanticweb.orgOntologyAPI

• All description tools that are present in the COMM can be used in Java applications

Currently supported (MPEG-7) description tools:• All visual low level descriptors (MPEG-7 part 3)‏• All media information descriptors (MPEG-7 part 5, clause 8)‏• Decomposition tools for

– Images (StillRegions, SpatialDecomposition, …) ‏– Videos (VideoSegments, TemporalDecomposition, …) ‏– Text (ASCIITextSegments, ASCIIDecomposition, …)‏

Semantic Annotation

with contributions by many west membersstaab/presentations/...kernel support vector machines hong et...

Documents