the k-space network of excellence: on the way to ‘semantic ...znalosti2008.fiit.stuba.sk ›...

The K-Space Network of Excellence: on the Wayto ‘Semantic’ Multimediato ‘Semantic’ Multimedia

Research Project

Presentation

V. Svátek, P. Berka, J. Nemrava, J. Petrák, P. Praks, M. Vacura

VŠE Praha

E. Izquierdo, C. Stewart

Queen Mary, University of London

Agenda

• Basic data about the project

• Main research directions

• Integration and dissemination activities

• Selected research achievements

Znalosti 2008, Bratislava

• Selected research achievements

K-Space project status

• FP6 IST project – Network of Excellence

• Started Jan 1, 2006, ends Dec 31, 2008

• Total EU funding 5.55 MEuro

• 14 partners (UK,FR,DE,AT,CH,IR,NL,GR,CZ)


– 7 universities

– 6 public research centres

– 1 content provider

• Website: http://www.kspace-noe.net

• See also flyer available in paper form

K-Space main focus

• Bringing the semantic web and multimedia communities closely together

• This is reflected in the structure of technical WPs– WP3 Content-based multimedia analysis

• Proceeding from low-level video/audio/image analysis to simpler semantic descriptors (esp. MPEG-7)

– WP4 Knowledge extraction


– WP4 Knowledge extraction

• From semantic object detection to application of high-level concepts (ontologies) and follow-up inference using rules

– WP5 Semantic multimedia

• Development of Semantic infrastructure relatively independent of media type, such as RDF stores and language extensions

– WP6 Integration of software tools

• Esp. wrt. joint participation in the TRECVid contest

Non-technical activities

• Academic exchanges of PhD students and researchers

• Temporary placements of PhD students to industrial environments

• Repository of teaching resources

• Organisation of an annual conference (called SAMT), summer school (called SSMS) and joint Master/PhD


summer school (called SSMS) and joint Master/PhD programme

• Dissemination of results using, among other, a six-monthly newsletter (available!)

• Joint contribution to standards (W3C, MPEG, JPSearch)

• Establishment of a permanent scientific society for this area, called SMaRT

Role of Czech partner

• The Czech partner, University of

Economics, Prague

– is involved in 3 out of 4 technical WPs

– coordinates 3 tasks (‘sub-WPs’)


• industrial placement of PhD students

• research on exploiting textual resources

complementary to multimedia

• dissemination via newsletter, poster and brochure

– contributes to 9 out of 16 technical tasks

Research achievements

• Follows a selection of research outcomes of K-Space, with special regard on activities in which UEP is heavily involved– Design of a system of multimedia ontologies

– Semantic merging of image regions using fuzzy rules

– Mining web resources complementary to sports video


– Mining web resources complementary to sports video

– Intra-frame similarity computation using LSI

– Two additional research streams are presented at this conference in the form of posters• Kolínsky, Nemrava, Svátek: Ukladanie výsledkov extrakcie

informácií do IPTC hlavičiek športových obrázkov

• Schenk, Petrák: Sesame RDF Repository Extensions for Remote Querying

Multimedia ontology

• The COMM (core ontology for multimedia) has been designed according to the following principles– Compliance with MPEG-7 standard

– Suitability for semantic web reasoning

– Adoption of patterns from foundational ontologies


– Adoption of patterns from foundational ontologies (DOLCE)

– Modularity, extensibility (for different media and domains)

• Used in the KAT annotation tool– Different MM objects can be manually or semi-automatically annotated according to COMM

Fragment of COMM upper layer

SocialObject

InformationEncodingSystem InformationObject Description

SituationAgent Particular InformationRealization

orderedBy

realizedByinterpretedByabout

satisfies

expresses

refersTo

Role Parameter

defines defines


conceivesOf

setting

refersTo

DigitalData StructuredDataDescription

StructuredDataInstantiation

satisfies

expresses

AbstractRegion

setting

about

Scalar Vector Matrix Polygon Rectangle

StructuredDataParameterStructuredDataRole

valuesplays

about

defines

Semantic region merging

• In image analysis, different regions are automatically identified, and assigned semantic classes according to their low-level properties

• UEP’s rule-based system NEST is used to


• UEP’s rule-based system NEST is used to evaluate the similarity between adjacent regions based on the initial segmentation and fuzzy labeling of regions

• Its results are then used for merging the most similar regions

Semantic region merging


Mining complementary resources

• For a given video (or image) there are typically various resources providing complementary information, e.g.– captions, subtitles

– speech transcripts

– web texts describing the same event or dealing with the same topic

• The application in the football domain attempts to


• The application in the football domain attempts to combine– analysis of audio/visual signal

– text detection and OCR in the video

– mining online match reports

– ontology-based reasoning (fuzzy description logics) over the concepts thus detected

Analysis of complementary resources

• Audio/video analysis

– 6 available detectors

• Crowd image

• Speech-Band Audio Activity

• On-Screen Graphics Tracking

• Motion activity


• Motion activity

• Close-up

• Field Line orientation



• Text detection

and OCR analysis

– mainly used to synchronise

video time with match time


• Tabular resources

– Basic Match

Information

• List of players, goals,

cards, etc.


cards, etc.

– Meta Information

• Location,

Attendance, Date,

etc.


• Information extraction from web-based

minute-by-minute reports



• Cross-media analysis using fuzzy

description logic


Intra-video similarity

• In the TRECVid contest,

a part of the task is to

detect the similarity of

frames within one

broadcast, so as to

Query formulation tool.


broadcast, so as to

identify e.g. news shots

from the same studio

• This is achieved by

applying a similarity-measuring method based

on LSI – Latent Semantic Indexing

Summary

• The K-Space project, during its first two

years, produced a high number of

research results that could immediately be

adopted by more applicative projects


• The third year, 2008, is focused on even

closer integration of research, and on

supporting a smooth start of the SMaRT

society, which is going to be a persistent

follow up to K-Space

Thanks for your attention


the k-space network of excellence: on the way to ‘semantic ...znalosti2008.fiit.stuba.sk ›...

Documents