The LinkedTV PlatformTowards a Reactive Linked Media Management System

Jan Thomsen, Rolf Fricke, Ali Sarioglu

Condat AG, Berlin, Germany

LiME Workshop 2016,

2016/05/29 Heraklion, Greece

What is LinkedTV?

• EU FP7 Project 2012/10 – 2015/03

• „Web and TV seemlessly interlinked“

• Some technologies involved:• Video Analysis, ASR

• Media Fragments

• Annotations and enrichment

• Second Screen

• Personlization

• Partners: CERTH (GR), CONDAT (DE), CWI (NL), EURECOM (FR) , Fraunhofer (DE), Noterik (NL), STI/MODUL (AT), rbb (DE), Institut for Sound & Vision (NL), UEP (CZ), U Mons (BE), U St. Gallen (CH)

The basic LinkedTV workflow

Video SubtitleMetadata

SegmentationConcept detectionFace recognitionASRSpeaker Detection

Media Fragment URIRDF GenerationAnnotationsNER

Manual Curation(EditorTool)Adding automaticenrichments

Second ScreenHbbTV AppNews App

User TrackingUser ProfilingFiltering

IngestionService

S&VAVRO‘sTKK

rbb

Analysis

ShotChapter

Segment.

VideoOCR

ConceptDetection

ASR

FaceDetection

tv2rdfAnnotation

NERD

rbbaktuell

Alchemy

dbpediaspotlight

ExtractivLupedia Open

Calais

Saplo

SemiTags

TextRazor

Wikimeta

Yahoo!

Zemanta

tvenrich

MediaCollector

IRAPI

Europeana

Tracking UserModeling

StreamingServer

HTMLPlayer

HbbTV

Filtering

SpringfieldMultiscreen

Toolkit

AttentionTracker

LUMO

SimpleLearner

EasyMiner

LUMELiFR

Recommender

LSF

SpeakerRecognition

FaceTracking

KeywordExtraction

ObjectRe-detection

THD

InBeat/GAIN

LISp-Miner

ContextTracker

Kinect

LinkedProfiler

EXMARaLDACreation

EditorTool

2ndScreen

LeapMotion

The Linked Media Ecosystem ofLinkedTV

LinkedTVPlatform

SRTSerilazer

TVAnytimeSerilazer

ExmaraldaSerilazer

LinkedCulture

EntityExpansion

TVNewsEnricher

Topic labeling

Linked Media Management in LinkedTV

Main functional features

• Generation of Audio Visual Profiles by analyzing video and audio tracks (e.g. chapter, scenes, objects)

• Generation of addressable Media Fragments (URIs)

• Generation of structured annotations describing these media fragments

• Named Entity Recognition and linking to LOD resources

• Enrichment

• Editing

Main technical features

• Ingestion of Media resources and associated files (e.g. subtitle files)

• Workflow Management

• Linked Data Repository

• REST API for interaction and development of Linked Media Applications for Web and TV (e.g. personalized Second Screen applications)

• Framework for easy integration of further services

• Tools and APIs for administration and configuration

Linked Media Management

Linked Media Management in context

Linked MediaManagement

Media Resources

Linked Data The Web

Content Management

Content DeliveryStreaming

Linked MediaApplications

Productionand

Publishing

The LinkedTV Platform: Version 1

• LinkedTV Platform Version 1 supports the general LinkedTV workflow

• Implements a very basic form of Linked Media Management

• Only a linear workflow

• Not scalable

• Not suitable for configuration of different workflows and smooth integration of further Linked Media services

• Not reliable and robust enough for production use

• Motivation for a redesign of the LinkedTV Platform

LinkedTV Platform Version 2 – Main Properties

• Network of indepedent microservices

• Each microservice acts as a consumer and/or producer

• Most microservices connect external REST services

• Operated by a message driven architecture (AMQP)

• Built on top of RabbitMQ

dispatcher.mediaresource.update_event_status(uuid, 'transferred', dispatcher.STATUS_FINISHED)

utilities.writelog(logfile, ' [Dispatcher] Transferred ...')

message = profile + ' ' + uuid + ' ' + video

ch.basic_ack(delivery_tag = method.delivery_tag)

channel.basic_publish(exchange='',

routing_key=queue_name_to,

body=message,

properties=pika.BasicProperties(

delivery_mode = 2,

))

utilities.writelog(logfile, "

[Dispatcher] INFO Sent to RabbitMQ Exchange %r:%r:%r" % (profile, message, queue_name_to))

The LinkedTV „Tube Map“

LinkedTV Platform Version 2 – Main Benefits

• Parallel processing

• Scalable and elastic: microservices can be multiplied and distributed among different servers

• Reliable: RabbitMQ takes care of message delivery

• Extendable: very easy to add new microservices without interfering with existent ones

• Easy to customize by selecting the microservices and defining routes

• In accordance to reactive design principles: Responsiveness, Resilience, Elasticity, Message Driven*

*reactivemanifesto.org

REST API data.linkedtv.eu

Summary and Discussion

• Linked Media needs new kind of „Linked Media Management Systems“

• The LinkedTV Platform is an example of such a system

• Still lacking a lot of features necessary for production use

• Does not extend to external services

• Currently only for the production part; to be extended into the publishing and consumption areas

• Next step: test runs for getting quantitave data

Questions

http://www.linkedtv.eu

jan.thomsen@condat.de

Thank you!