remixing media on the semantic web (iswc2014 tutorial) pt 2 linked media: an approach to online...
DESCRIPTION
The second session looks at how using Linked Data principles for media fragment annotation publication and retrieval (Linked Media) can enable online media fragment re-use: Introducing the Linked Media principles Publishing Linked Media using dedicated multimedia RDF repositories Retrieval of media resources that illustrate linked data concepts Using the Linked Data graph to find relevant links between distinct media assets (examples with SPARQL) Retrieval of links between annotated media to enable topical browsing (using the TVEnricher service) Examples of Linked Media at scale: VideoLyzard and HyperTEDTRANSCRIPT
1
LinkedMedia:An approach to online media re-use
Lyndon NixonMODUL University [email protected]
RE-USING MEDIA ON THE (SEMANTIC) WEBISWC2014 Tutorial, Riva de Garda, Italy, October 20 2014
20.10.14 Slide 2 of 50
Agenda
• Session 1: Media fragment specification and semantics• Summary: Introduce the W3C Media Fragment URI specification
and the Open Annotation model. Highlight how media fragments can be annotated using NER tools.
• Session 2: Linked Media principles• Summary: Introduce the Linked Media principles. How to
publish Linked Media in RDF and how to retrieve media enrichments. Illustration with Linked Media applications.
• Session 3: User experience driven design of Linked Media applications• Summary: Present the Web and TV convergence. Describe
LinkedTV experience via two innovative applications.
2
20.10.14 Slide 3 of 50
2/3: LinkedMedia
• Introducing the Linked Media principles
• Publishing Linked Media• LinkedTV Platform (Virtuoso)
• Retrieval of Linked Media• Enrichment of media (LinkedTV)• Browsing and linking media
(VideoLyzard, HyperTED)
3
20.10.14 Slide 4 of 50
Linked Media
20.10.14 Slide 5 of 50
„It is growing at more than 20% per annum, fuelled by increased demands for new programming and the huge saving it represents compared with shooting new footage. Interactive technology and the Internet will further contribute to the growth of the market as it makes stock footage cheaper and easier to locate and license.“ - http://moneyam.uk-wire.com/cgi-bin/articles/200201020827103514P.html
Why is Online Media important?
20.10.14 Slide 6 of 50
Why Semantic Media?
20.10.14 Slide 7 of 50
Why Linked Media?
20.10.14 Slide 8 of 50
Why Linked Media?
20.10.14 Slide 9 of 50
Why Linked Media?
20.10.14 Slide 10 of 50
Why Linked Media?
From Lyndon Nixon, „The importance of Linked Media to the Future Web“slideshare.net/linkedtv/www-linked-media-keynote
20.10.14 Slide 11 of 50
Linked Media Principles
1. Web media descriptions need a common representation of media structure
2. 2. Web media descriptions Web media descriptions need a common need a common representation of media representation of media contentcontent
3. 3. Web media descriptions need to use a Web media descriptions need to use a media ontology which supports description media ontology which supports description of both the structure and content of mediaof both the structure and content of media
20.10.14 Slide 12 of 50
Linked Media Principles
4. 4. The descriptions of The descriptions of media in terms of media in terms of common common representations of representations of structure and content structure and content are the basis for deriving are the basis for deriving links across media on links across media on the Web (Linked Media) the Web (Linked Media)
20.10.14 Slide 13 of 50
Linking Media
Prime Minister
South Africa President
Mozambique
20.10.14 Slide 14 of 50
PublishingLinked Media
20.10.14 Slide 15 of 50
Linked Media is Linked Data
Good news: Linked Media can be published as Linked Data!
•The media resource has a globally (Web wide) unique identifier•Metadata about the media resource can be accessed via its identifier•Media identifier can‘t be = media locator!
20.10.14 Slide 16 of 50
LinkedTV Platform
Web administration interface & REST API athttp://api.linkedtv.eu
20.10.14 Slide 17 of 50
Media Resource view
20.10.14 Slide 18 of 50
Media Metadata view
http://data.linkedtv.eu is the base of the RDF graph of LinkedTV content, and via Virtuoso, every LinkedTV instance URI can return HTML or RDF, e.g.http://data.linkedtv.eu/mediaresource/8a8187f2-3fc8-cb54-0140-7dccd76f0001
20.10.14 Slide 19 of 50
Media Metadata view
Media fragments of a media resource:
http://data.linkedtv.eu/mediaresource/8a8187f2-3fc8-cb54-0140-7dccd76f0001/mediafragment
20.10.14 Slide 20 of 50
Media Metadata view
Annotations of a media fragment:
http://data.linkedtv.eu/mediafragment/3d5b11e0-f6df-11e3-b0fe-005056a7235c%23t=1,299/annotation
20.10.14 Slide 21 of 50
Linked Mediaretrieval
20.10.14 Slide 22 of 50
SPARQL endpoint
SPARQL queries allow us to connect metadata across the Linked Media store AND to connect it to other metadata outside in the Linked Data cloud.
Linked Media queries courtesy LinkedTV deliverable 2.4 „Annotation and retrieval module of media fragments“ (Jose Luis Redondo Garcia & Raphael Troncy) available from:http://de.slideshare.net/linkedtv/annotation-and-retrieval-module-of-media-fragments
20.10.14 Slide 23 of 50
SPARQL queries
Get all Shots or Chapters for a media resource.
20.10.14 Slide 24 of 50
SPARQL queries
Return all entities of type nerd:Person in the annotation of a media resource.
20.10.14 Slide 25 of 50
SPARQL queries
Return all entities occurring within a temporal boundary of a media resource.
20.10.14 Slide 26 of 50
LinkedTV
20.10.14 Slide 27 of 50
Why Linked Television?
40% of TV viewers are using a companion device alongside the TV program.*
* J. Abreu, P. Almeida, B. Teles, and M. Reis. Viewer behaviors and practices in the (new) television environment. In Proceedings of the 11th European Conference on Interactive TV and Video, EuroITV '13.
http://www.linkedtv.eu
Ever saw something on TV
and wanted to know more about it, but didn‘t even know how to search for
it?
20.10.14 Slide 28 of 50
LinkedTV Technology
“...schilderij van Jan Sluijters....”
Video object and word detection
dbpedia.org/resource/Jan_Sluyters
Connection to concepts
Paintings by Jan Sluijters
Selection of related concepts
Selection of related content
Presentation engine
20.10.14 Slide 29 of 50
Related concepts
Additional information, e.g. biography of artist style of painting
Related information, e.g. artists from same
period paintings in similar style related styles
Paintings by Jan Sluijters
Expansion of related concepts
Jan Sluijters
has art style
luminism
Leo Gestel
Piet Mondriaan
has art style
has art style
The use of Linked Data in the identification of concepts in LinkedTV means we can expand concepts along different facets, i.e. allow users to explore in terms of their different interests in a given concept.For example, for an artist like Jan Sluijters, LinkedTV can link into:
20.10.14 Slide 30 of 50
Related content
LinkedTV provides enrichment services which provide recommendations for related online content (Web pages, images, audio, video) pertaining to the concepts in the TV program:– Fresh Social Web content coming
from e.g. Twitter and Facebook– User Generated content coming
from e.g. Flickr and YouTube– Whitelist content coming from
partner Websites like public broadcasters in Germany or cultural heritage archives in the Netherlands
– Extendable and configurable by source and media type
Linking to related content
… and their digital images
20.10.14 Slide 31 of 50
LinkedTV „enrichment“
Base enrichments: title, thumbnail (poster), description (abstract)
Information cards: set of properties and values according to the entity type
Linksets: links to online content determined by queries over Web content sources using a group of entities as search term•Linksets can be split along enrichment dimensions•Dimensions are distinct aspects of interest to viewers•They map to different queries & services in LinkedTV
20.10.14 Slide 32 of 50
LinkedCulture enrichment
TKK Video
Chapter segmentation + tagging as „Art Object“
TV2RDF incl. Art Object annotation
RelatedTKK chapters(Solr)
Relatedart objects (Europeana)
RelatedWhite List media(IRAPI)
TVEnricher (TKK configuration)
Editor Tool LinkedTV Player
Entity Proxy
20.10.14 Slide 33 of 50
Art objects in TKK episodes
20.10.14 Slide 34 of 50
Art object: semantic model
http://data.linkedtv.eu/object/avro/8a8187f2-3fc8-cb54-0140-7dccd76f0001/2138
A silver tea jar
RDF Is-a Container http://vocab.getty.edu/aat/300045611
CRM Consists-of Silver http://vocab.getty.edu/aat/300010975
VRA locationCreationSite
Friesland http://www.geonames.org/2755812
DCT temporal Start: 1690, End: 1742
20.10.14 Slide 35 of 50
Mapping to Europeana API
http://data.linkedtv.eu/object/avro/8a8187f2-3fc8-cb54-0140-7dccd76f0001/2138
A silver tea jar
RDF Is-a Container
CRM Consists-of Silver
VRA locationCreationSite
Friesland
DCT temporal Start: 1690, End: 1742
what:(container+OR+houder+OR+bak+OR+tank+OR+blik) proxy_dc_format:zilver
where:Friesland
YEAR:[1690+TO+1742]
20.10.14 Slide 36 of 50
Enrichment results
Friesian silver from 1690 to 1742 to 1742
20.10.14 Slide 37 of 50
LinkedNews enrichment
RBB Video
Chapter segmentation + tagging as „News Item“
Named Entity Expansion over news items
Relatednews items (Solr)
Relatednews articles (TVNews-Enricher)
RelatedWhite List media(IRAPI)
TVEnricher (RBB configuration)
Editor Tool LinkedTV Player
Entity Proxy
20.10.14 Slide 38 of 50
Entity Proxy
Fill information cards for entities giving values for their most relevant properties.
20.10.14 Slide 39 of 50
TVNewsEnricher
Returns news media sources related to given entity sets, powered by Google CSE
20.10.14 Slide 40 of 50
IRAPI: Web media crawler
Extract descriptions of media on websites so that related media can be found
Video
TitleDescription
Entities, e.g. http://dbpedia.org/resource/Edward_
Snowden
20.10.14 Slide 41 of 50
VideoLyzard
20.10.14 Slide 42 of 50
Climate Change Portal
42
20.10.14 Slide 43 of 50
VideoLyzard
43
http://link.weblyzard.com/video-showcase
Videofragments in search results
Videofragment playback
20.10.14 Slide 44 of 50
VideoLyzard
20.10.14 Slide 45 of 50
VideoLyzardMore Negative More Positive
20.10.14 Slide 46 of 50
HyperTED
Content courtesy EURECOM (José Luis Redondo García, Raphael Troncy, Mariella Sabatino & Pasquale Lisena)
20.10.14 Slide 47 of 50
1984
.com
2006
CHAPTERS
2014
HOT SPOTS
ENTITIES
RELATED TED’S
CHAPTERS
THE MYSTERIOUS
FIELD OF ENGINEERING
SYSTEMS
UNDERSTANDING
ENVIRONMENT:
A SYSTEM APPROACH
SYSTEMS PRACTICE: MANAGING SUSTAINABI
LITY
COURSES
20.10.14 Slide 48 of 50
HotspotsCluster video chapters which share similar topics
20.10.14 Slide 49 of 50
Architecture
49
20.10.14 Slide 50 of 50
Demo
http://linkedtv.eurecom.fr/HyperTED
20.10.14 Slide 51 of 50
MediaMixer community portal
Introduction to all technologies at community.mediamixer.eu/technology
Updated with latest materials on all Media Mixer topics:
Technology use cases Demonstrators Tutorials Presentations Software Specifications
http://community.mediamixer.eu
20.10.14 Slide 52 of 50
VideoLecturesMashup
20.10.14 Slide 53 of 50
MediaMixer use case: VideoLecturesMashup
20.10.14 Slide 54 of 50
Video fragment creation
Fragments were created based on the slide synchronisation timeline.
Transcripts (auto-generated by speech-to-text technology where necessary) were parsed and split across fragments.
… there are three Kingdoms of Life, Bacteria, Archaea and Eukaryota...
20.10.14 Slide 55 of 50
Video fragment annotation
Fragments were then annotated by extracting topics from their textual metadata (slide OCR or speaker transcription).
Topics are connected to a global knowledge model (DBPedia).
Video Fragment (4:41-5:12)
Archaea
20.10.14 Slide 56 of 50
Video fragment management
Annotations are managed in a separate metadata store.
The store provides a semantic query endpoint returning lists of video fragments matching a query topic (including semantically related topics)
Archaea
Acidiplasma „type“ relation
Video Fragment (4:41-5:12)
20.10.14 Slide 57 of 50
Video fragment playback
The front end uses HTML5 or Flash. Both codebases are extended to support video fragment playout.
Individual playback can be modified to linear or non-linear channels (for e.g. a TV or mobile video experience)
20.10.14 Slide 58 of 50
VideoLecturesMashup - demo
http://mediamixer.videolectures.net
20.10.14 Slide 59 of 50
20.10.14 Slide 60 of 50
20.10.14 Slide 61 of 50
20.10.14 Slide 62 of 50
20.10.14 Slide 63 of 50
Reference
20.10.14 Slide 64 of 50
SPARQL RDF query language
RDF data is a „labeled, directed graph“ without any fixed vocabulary (this is defined in RDF Schema). A query language for RDF needs to support this data model:•Tranversing paths in a RDF graph•Being independent of any defined vocabulary•Able to query over the data or the schema
SPARQL („sparkle“) is a W3C standard supported in most RDF tools•SELECT... WHERE.... constructions like in SQL•Path expressions•Additional keywords for more query expressiveness
20.10.14 Slide 65 of 50
Examples: triple patterns
The basic idea in SPARQL is to match graph patterns against RDF graphs.To understand graph patterns we must first define triple patterns:• A triple pattern is similar to a triple (in RDF) but with one or more variables in the place of a RDF resource (URI or literal)
Triple: dbpedia:Lou_Reed foaf:givenName „Lewis Allen Reed“ .
Triple pattern: dbpedia:Lou_Reed foaf:givenName ?name .
?name is the variable. If a RDF graph with the above triple is queried with the below triple pattern, then in the SPARQL results the variable ?name would be ‚bound‘ to the value „Lewis Allen Reed“.
• A SPARQL query result is a set of bindings for the variables appearing in the SELECT clause, based on matching RDF resources to variables in triple patterns in the WHERE clause. .
20.10.14 Slide 66 of 50
Examples: conjunctive and disjunctive query
In graph patterns, several triple patterns are listed within braces { ... } and these are interpreted conjunctively:
Ex: { ?what tech:noOfWheels „4“ . ?what tech:minSpeed „180“ . }
The variable ?what will only be bound to resources which BOTH have 4 wheels AND a minimum speed of 180.
You can also join results from distinct graph patterns using the UNION keyword. Note that result sets from graph patterns and from UNIONs are different, since UNION works disjunctively:
Ex: { ?what tech:noOfWheels „4“ . } UNION { ?what tech:minSpeed „180“ . }
20.10.14 Slide 67 of 50
Examples: FILTER, ORDER BYSPARQL has many other keywords. FILTER restricts variable bindings returned in query results to those for which the filter expression evaluates to true. A filter applies to solutions over the entire graph pattern it is contained in.
Ex: { ?what tech:noOfWheels „4“ . ?what tech:minSpeed ?speed .
FILTER ( ?speed > 170 ) }
The ORDER BY keyword determines the sequence of query results returned according to a sort on the referenced variable‘s bindings, ascending by default:
Ex: { ?what tech:minSpeed ?speed . }ORDER BY ?speed
Or ORDER BY DESC(?speed)
20.10.14 Slide 68 of 50
Query for media fragments: SPARQL-MMResearch work in progress: extending SPARQL to Media Fragments by adding spatio-temporal filter and aggregration functions.
Relation Function Aggregation Function
Spatial mm:rightBeside mm:spatialIntersection
mm:spatialOverlaps mm:spatialBoundingBox
… …
Temporal mm:after mm:temporalIntersection
mm:temoralOverlaps mm:temporalIntermediate
… …
Combined mm:overlaps mm:boundingBox
mm:contains mm:intersection
Courtesy Thomas Kurz (Salzburg Research) & MICO EU projecthttp://demos.mico-project.eu/sparql-mm/sparql-mm/demo/index.html