producer · (pre-production, production & post-production tools) task no. t3.3 task title...

The Framework Programme for Research & Innovation

Innovation actions (IA)

Project Title:

PeRsOnalized DocUmentary Creation based on Automatically Annotated Content

PRODUCER Grant Agreement No: 731893

[ICT-21-2016] Support technology transfer to the creative industries

Deliverable

D3.3 – Post-production related tools

Deliverable No. 3.3

Work package No.

WP3 Work package Title Platform Toolset Development (Pre-production, Production & Post-production Tools)

Task No. T3.3 Task Title Interactive-Enriched Video Creation, 360° Video Playout, Second Screen Interaction & Social Recommendation & Personalization Tools

Lead beneficiary Fraunhofer FOKUS

Dissemination level PU

Nature of Deliverable Report

Delivery date 31-12-2017

Status D: final

File Name: [PRODUCER] D3.3: Post-production related tools.docx

Project start date, duration 01 January 2017, 18 months

This project has received funding from the European Union’s Horizon 2020 Research and innovation programme under Grant Agreement n°731893.

Version of 2017-12-31


page 2

Authors List

Leading Authors (Editor)

Full name Initials Beneficiary

Name

Contact email

Louay Bassbouss LB FOKUS [email protected]

Christian Fuhrhop CF FOKUS [email protected]

Giorgos Mitsis GM ICCS [email protected]

Reviewers List

List of Reviewers (in alphabetic order)

# Surname Initials Beneficiary Name

Contact email

1. Christopher Ververidis

CV HIT [email protected]

2. Nikos Kalatzis NK ICCS [email protected]

Document history

Version Date Status Modifications made by

0.1 17/11/2017 Table of Contents Christian Fuhrhop - FOKUS

0.2 17/11/2017 Interactive Annotation Tool Christian Fuhrhop - FOKUS

0.3 30/11/2017 Second Screen Framework Louay Bassbouss - FOKUS

0.4 07/12/2017 360° Video Playout Louay Bassbouss - FOKUS

0.5 15/12/2017 Social Recommendation & Personalization Tools

0.6 18/12/2017 Introduction, Summary and post-production workflow added

Christian Fuhrhop - FOKUS

0.7 20/12/2017 Review Christopher Ververidis - HIT

1.0 22/12/2017 Final version Louay Bassbouss - FOKUS



page 3

Abbreviation List

Abbreviation Definition

CA Consortium Agreement

CoA Coordination Agreement

DoW Description of Work

EC European Commission

IPR Intellectual Property Rights

NDA Non-disclosure agreement

PO Project Officer

QA Quality Assurance

R&D Research and Development

WP Work Package



page 4

Acknowledgement: The research leading to these results has received funding from the European Union's Horizon 2020 Programme (H2020-ICT-2016, call ICT-21-2016) under grant agreement n° 731893.

Disclaimer: This document does not represent the opinion of the European Community, and the European Community is not responsible for any use that might be made of its content.

This document may contain material, which is the copyright of certain PRODUCER consortium parties, and may not be reproduced or copied without permission. All PRODUCER consortium parties have agreed to full publication of this document. The commercial use of any information contained in this document may require a license from the proprietor of that information.

Neither the PRODUCER consortium as a whole, nor a certain party of the PRODUCER consortium warrant that the information contained in this document is capable of use, nor that use of the information is free from risk, and does not accept any liability for loss or damage suffered by any person using this information.



page 5

Executive Summary

This deliverable covers the PRODUCER tools that aid documentary film producers in the post-production process.

It supplements deliverables D3.1 and D3.2, which cover the PRODUCER tools for the pre-production and the production phases respectively. The structure of the three deliverables, covering the complete production workflow, is intentionally kept similar, so that all three deliverables together can be considered one descriptive document of PRODUCER tools.

Post-Production covers, in general, all aspects of documentary making that occur after the shooting of the initial video material. While pre-production is typically considered to include non-technical aspects as well (such as securing shooting locations, obtaining permits and casting or hiring personal), the term post-production usually only covers the technical issues of film productions. Distribution and advertising are not considered part of post-production, although technical aspects that relate to it (such as the production of a trailer) are.

In PRODUCER, the producer is supported in the creation of the documentary by four tools:

1. Interactive enriched video creation tool This supports the inclusion of supplementary (and interactive) information as part of the documentary for traditional and 360° video content

2. Second Screen Framework This supports the inclusion of supplementary information or control mechanism for a documentary on a second device (typically tablet or smartphone) for traditional and 360° video content without cluttering the main presentation screen.

3. Playout for 360° video content This enables producers to provide 360° content to TVs for home viewers

4. Social Recommendation and Personalization Tools This allows users to provide feedback about the documentary by commenting and rating it. It also allows providing recommendations to the user, based on personal preferences.

The deliverable provides descriptions of all tools and adds, where appropriate, their placement in the overall architecture, their placement in the production process, a reference to identified user requirements, their key components, APIs and user interfaces, access information and a description on how the tool was tested in the PRODUCER project.



page 6

Table of Contents

1 Introduction 9

2 Post-production workflow 9

3 Interactive Enriched Video Creation Tool 11

3.1 Overview 11

3.2 Getting started 11

3.2.1 NLV edit screen 12

3.3.1 Tutorial 1: Placing and moving a marker 14

3.3.2 Intermezzo: Using the equirectangular view 18

3.3.3 Intermezzo: Other basic marker types 19

3.3.4 Tutorial 2: Interacting elements and fixed view markers 20

3.3.5 Intermezzo: Coordinates relative to scene or screen 25

3.3.6 Tutorial 3: Controlling the view 26

3.3.7 Intermezzo: Following objects 28

3.3.8 Tutorial 4: Navigating the timeline and chaining commands 30

3.3.9 Tutorial 5: Overlays 32

3.4 Functional Description 33

3.5 Mapping of tool’s functionality to the user requirements identified 34

3.6 Advantages and added value to the PRODUCER Platform 34

3.7 Architectural Diagram and placement in the overall architecture 35

3.8 Key Technologies 36

3.9 Code structure and availability 36

3.10 Graphical Interfaces 36

3.11 How to access the tool 37

3.12 Sample Use Cases (Workflows) 37

3.13 Datasets used for tool testing 37

4 360° Video Playout 37




4.4 Architectural Diagram and Placement in the overall architecture 38



4.7 Programming Interfaces 41



page 7



5 Second Screen Framework 43




5.4 Architectural Diagram and Placement in the overall architecture 45



5.7 Programming Interfaces 47

5.8 How to access the framework 47


6 Social Recommendation and Personalization Tools 54


6.1.1 Personalization 54

6.1.2 Relevance Feedback 55

6.1.3 Social Recommendation 56



6.4 Architectural Diagram & Placement in the overall architecture 59



6.7 Implementation 60

6.7.1 Graphical Interfaces 60

6.7.2 Algorithms 66

6.7.2.1 Initialization 66

6.7.2.2 Multimedia Content Indexing 67

6.7.2.3 Relevance Feedback 68

6.7.2.4 User Profiles Update 70

6.7.2.5 Recommendation mechanisms 70

6.7.2.6 Target group recommendations 72

6.8 How to access the tool 72


6.9.1 Audience member workflow 73



page 8

6.9.2 Content creator workflow 74


7 References 76



page 9

1 Introduction

This deliverable covers the PRODUCER tools that aid documentary film producers in the post-production process.

It supplements deliverables D3.1 and D3.2, which cover the PRODUCER tools for the pre-production and the production phases respectively.

Post-Production covers, in general, all aspects of documentary making that occur after the shooting of the initial video material. While pre-production is typically considered to include non-technical aspects as well (such as securing shooting locations, obtaining permits and casting or hiring personal), the term post-production usually only covers the technical issues of film productions. Distribution and advertising are not considered part of post-production, although technical aspects that relate to it (such as the production of a trailer) are.

In PRODUCER, the producer is supported in the creation of the documentary by four tools:

1. Interactive enriched video creation tool This supports the inclusion of supplementary (and interactive) information as part of the documentary for traditional and 360° video content

2. Second Screen Framework This supports the inclusion of supplementary information or control mechanism for a documentary on a second device (typically tablet or smartphone) for traditional and 360° video content without cluttering the main presentation screen.

3. Playout for 360° video content This enables producers to provide 360° content to TVs for home viewers

4. Social Recommendation and Personalization Tools1 This allows users to provide feedback about the documentary by commenting and rating it. It also allows providing recommendations to the user, based on personal preferences.

2 Post-production workflow

Most PRODUCER tools are located near the end of the traditional post-production workflow.

While this is not a rigid rule, the PRODUCER tools assume the documentary to be in its final, broadcast or cinema ready version. Especially the interactive features (whether on primary screen by the interactive enriched video creation tool or on second screen using the second screen framework) are synced to the video timeline. While both can be adjusted later, the optimal workflow places them at least after the documentary is cut, although, conceptually, they can be utilized in parallel with sound editing and related activities (although Foley work is rare on documentary productions).

1 This is not considered part of the post-production process in the traditional use of the term, but is an important feature, especially with social media being a significant part of a documentaries reception and success.



page 10

An article by Elliot Grove,2 the founder of Raindance Film Festival and the British Independent Film Awards, lists 13 steps of post-productions. While the article focuses on the production of fiction, most of the steps, (except, as mentioned, Foley work) apply to documentaries as well.

The table below gives an overview of these 13 steps and the order in which the PRODUCER tools typically follow in the post-production schedule.

1. Pick an editing format

2. Hire a picture editor

3. Hire a sound editor

4. Do ADR

5. Do Foley

6. Secure music3

7. Do re-recording/the mix

8. Get an Music&Effects Only track

9. Get your titles

10. Get a Digital Cinema Package

11. Get a dialogue script

12. Get a campaign image

13. Get a trailer

14. Enrich video with interactivity

15. Add second screen functions

16. Provide 360° viewing capability for TV viewers

17. Allow feedback and social media activities

The video enrichment and second screen features can be, depending on production schedule, added as early as after step 2 (in the original list of 13 steps, the step named “hiring a picture editor” implicitly includes the work of the person hired), from the moment the production enters 2 https://www.raindance.org/the-13-steps-of-post-production/ 3 The only non-technical step in the traditional pre-production process



page 11

the status “picture locked”. The workflow however shows the most typical workflow.

3 Interactive Enriched Video Creation Tool

3.1 Overview

This section presents the interactive enriched video creation tool in the form of a tutorial/handbook, describing the main functions and features of the tool, starting with simple controls and leading with small steps toward complex scenarios.

3.2 Getting started

Open the URL for the annotation tool and log in.

For the producer project, please log in as “[email protected]”.

(The required password is available within the project, but will not be documented in this public deliverable.)

The first page is the “Dashboard”. This allows you to administrate multiple projects.

For this tutorial, we will ignore any projects that might be already here. Click on “Add Project”.

The first step is to upload a 360° video. For this, select ‘Upload’, enter a descriptive name for the video in the “Provide a name for the new video” textbox, turn the “Activate 360° support?” switch on and press the upload icon ( ) to upload a local file.

Should the file already be available online, select ‘URL’ instead of ‘Upload’ and provide the URL of the 360° video in the “Provide the URL for the source video” textbox.

After pressing the “Next” button, you need to provide a project name in the “Name for the new



page 12

project” text field and press “Save”.

The dashboard should now show your project.

To annotate the video, click on the edit ( ) button.

If the video doesn’t play and gives an error message, it sometimes helps to wait a while before continuing. “Uploading” a video starts the upload in the background. Depending on the size of the video and the available bandwidth, uploading may take some time. Often the error message only indicates that the video has not been fully uploaded.

Try to play the video.

3.2.1 NLV edit screen

The edit screen should look something like this:



page 13

The main elements are:

Main menu

The primary use is the selection of the “Dashboard” menu item to return to the dashboard screen

Video window

The 360° video view, which is used for the placement of the individual annotation and marker elements.

Models

This box provides access to the various annotation and interactive elements that can be placed on the video.

Settings

This box allows access to the parameters of the individual elements. When an element is selected in the ‘Marker’ box, its parameters are shown here and can

be modified.

Marker

This provides access to the individual markers that are placed on the video and their timing information (keyframes for the markers position and when they become visible or invisible). Clicking on the marker name brings up the parameters for the marker in the Settings box



page 14

3.3.1 Tutorial 1: Placing and moving a marker

In this tutorial we will place a simple marker pin in the scene and make it move with the marked object.

Watch the video and select an object to be marked. For the tutorial we select this little bamboo stick standing in the snow.

Go to the moment in the video where the object first becomes visible and pause there.

Select “nlv-icon” from the “Models” menu and drag the element over the video.

In the “Marker” box, you now see the marker and two time entries.

The first one denotes the creation of the marker (which is usually at the beginning of the video) and the time where it becomes visible. The eye icon ( ) denotes whether an element is visible or invisible. A greyed out eye indicates that the element becomes (or stays) invisible at that time, a black eye indicates that the element becomes (or stays) visible.



page 15

As numbered names can become confusing after a while and the pin is too large for the scene, we can click on the “Marker 1” text to bring the parameters for this marker into the “Settings” box. (Clicking on the element name in the “Marker” box also expands or hides the time entries for the marker.)

The “Settings” box provides access to all parameters of an element.

At the moment, we are only interested in three of them.

In the “Name:” text field, we enter “Bamboo Pole” to get a more meaningful name for the marker.

From the “Icon:” pull-down menu, we select “flag”, as this fits the function of the pole better than the “info” icon.

And as it is just a minor element in the scene, we set “Size:” to “.5” to make the icon smaller.

As a result, the video window now looks like:



page 16

As the bamboo pole is moving in relation to the view, the icon marker needs to be moved accordingly. This is done by setting keyframe positions.

We play the video and stop at some later point when the object we want to anchor our icon to has moved. We then grab the icon and move it to the new position.

The marker is moved through the scene in a linear path between the keyframe positions.

In most cases the best approach will be to set the position of the icon marker when the object enters the scene and one where it leaves the scene. And then watch the movement of the marker between these points and adjust the position when it moves away most from the object it marks.

Results are usually best when the video is paused for moving the marker. It is, however, possible to move the marker while the video is running. For this, it is important to know that a new keyframe position will be created every time the left mouse button is released. So you



page 17

grab the marker, move it with the mouse along the desired path and every time you want to set the current position, you release the object, ‘dropping’ a keyframe position at the current position.

Markers exist by default during the whole video. As in our example, the bamboo pole is entering the frame at 2:08 and leaves the frame at 2:23, we need to make sure that the marker is only visible during this time. For the appearance of the marker, this happens automatically. The marker exists from time 0:00, but the visibility icon is greyed (denoting a non-visible marker) and only becomes black at 2:08 (shown by the black visibility icon).

But the marker would remain visible at its final positon (at 2:23 in the video), so we need to click on the visibility icon to turn the marker visibility off when the bamboo pole gets out of view.

Markers can also re-appear.

In this example, we are passing multiple bamboo poles, so we can re-use the marker and also attach it to the next bamboo pole that is coming along.

In this example, the first bamboo pole enters the scene at 2:08 and leaves at 2:23. A second pole appears at 2:33 and leaves at 2:29, resulting in this timing list for the marker icon.



page 18

3.3.2 Intermezzo: Using the equirectangular view

So far, we have been using the ‘user’ view of the video, which provides a movable ‘window’ into the 360° scene. While this is the view the user will have and the most ‘natural’ looking for 360° videos, it is sometimes cumbersome to follow an object in this view, as, unless the object always stays in the same view, it is necessary to adjust the view from time to time to keep the tagged object in sight. There is also the risk that, when ‘grabbing’ an area of the scene to move It around, to accidentally grab the marker and, instead of moving the viewport, moving the marker.

NLV 360° also allows marker positioning in equirectangular view. For this, press the round

button at the lower right of the video window ( ). This changes the view from the viewport to the unmapped equirectangular source video.

While this view is distorted and a bit unusual, it shows the complete source video, so all visible objects stay within the view and can be easier followed and tagged.

Note that the equirectangular view is primarily for setting marker positions and overlays are not adjusted specifically for this view. So a marker of a given size will have the same size all over the equirectangular frame and keep the same shape, even though the underlying video is distorted and the apparent size of elements in the scene depends on the position in the frame.



page 19

To check “how it looks for a user” it is recommended to switch to the 360° view regularly. However, for the setting of marker positions, the equirectangular view is a useful tool.

3.3.3 Intermezzo: Other basic marker types

In the first tutorial, we used a pre-defined marker type ‘nlv-icon’. This provides a predefined list of (currently) ten icons: info, room, star, audiotrack, landscape, done, clear, explore, flag and share.

Info Room Star Audiotrack Landscape Done Clear Explore Flag Share

The marker type ‘nlv-shape’ provides a second set of pre-defined marker shapes: Square, Rectangle, Circle, Oval, Triangle-Up, Triangle-Down, Triangle-Left, Triangle-Right, Triangle-Top-Left, Triangle-Bottom-Left. Triangle-Top-Right, Triangle-Bottom-Right, Trapezoid, Parallelogram, Star, Pentagram, Hexagram, Diamond-Square, Diamond-Narrow, Talk Bubble.

As these shapes are ‘drawn’ over the video (as opposed to the icons, which are based on bitmaps), it is possible to change opacity and colour for shape markers.

Some examples:

Triangle-Down, blue Circle, red Parallelogram, green Talk Bubble, white,

opacity 60% over sky

background

Short info about objects in the scene can also be provided with the ‘nlv-text’ marker.



page 20

Text markers allow the specification of a text colour in RGB hex format (e.g. #ff0000 for red, #ffff00 for yellow, #808080 for grey and so on). For better readability, it is also possible to specify the background colour for the text in the same format.

Text markers are best suited for labelling objects in a scene or to provide navigation hints (like “Click me”). Longer, descriptive texts should be avoided. In most cases it is preferable to render descriptive texts as images.

3.3.4 Tutorial 2: Interacting elements and fixed view markers

In Tutorial 1, we created a marker that followed a bamboo pole.

So far, the marker didn’t do much but follow bamboo poles in the video. There was no interaction and no useful information for the viewer. To add this, we prepare a little info graphic and uploaded it to any web server, so we can access it via an URL.

We then select the ‘nlv-image’ marker and drag into the video frame, where it first appears with a default image (the red marker in the screenshot below).



page 21

We select that marker in the ‘Marker’ box to access its parameters in the ‘Settings’ box.

Here we provide a readable name for the marker (“Bamboo Text”), put the link to the image in the “Src:” field and adjust the size of the image (by putting “5” in the “Size:” field).

The image now appears in the video like this:



page 22

While this works, it is very intrusive for the viewer to have a large text/image banner on screen all the time. To avoid this, we link the visibility of the image to clicking on the original marker.

To achieve this, we need the internal code for the image marker.

We find that when we click on the marker in the “Marker” box and then copy the string from the Uuid: text field (in this case “DC1F9F0D-053C-4DA5-B05C-6340EE362E12”), which represents the internal ID of the object.

We can then select the interaction object (the icon marker named “Bamboo Pole”), go to “Settings” and scroll down to the “OnClick:” field, where we select “toggleVisibility” as the action and paste the UUID of the image marker below it.

To see the effect more clearly, we go back to the “Marker” box, select the “Bamboo Text“ marker and switch of its visibility for all keyframes, making the image initially invisible for the whole video.

When watching the video, it is now possible to switch the image with the descriptive text on and off by clicking on the icon marker.



page 23

Previewing interactive videos: When trying to interact with markers while in edit mode, this only works when the video is paused and a keyframe is selected. If the video is running or paused on a frame that is not a keyframe for the selected element, the editor assumes that you are trying to position the selected marker on this frame and create a keyframe, which can cause confusion and move a marker when you don’t want it to. The best way to test interactive elements is to go into preview mode. For this, select the “Dashboard” button on the upper right

and then select the “Share” icon ( ) for your project (the middle icon under “Actions”). This will give you a preview link that you can click on to see the video like a user would see it,

allowing interaction, but not editing. To get back to editing, select the edit icon ( ) again.

While the interaction works, there are some apparent issues that should be fixed.

First of all, the descriptive image can only be switched when clicking on the marker icon. When the image has been turned on, but the marker has vanished from view, there is no way for the user to remove the image again.

To fix this, we select the “Bamboo Text“ marker and add its own Uuid into the text field below its own “OnClick:” function and select the “toggleVisibility” action here as well.

As a result, the user can remove the descriptive text image at any time by clicking on it.

At the moment, there is no indication that the user can interact with the flag. As an initial fix, we can select the “Bamboo Pole” marker in the “Marker” box, go to the “Settings” box and scroll down to the “OnHover” item, select the “tooltip” option and enter some text (“Click for info”) below.



page 24

When the video is played now, moving the cursor over the flag icon marker, the tooltIp text is displayed, giving the user an indication that clicking on the icon will perform some action.

While this only appears when the user moves the cursor over the icon and there is no other indication that this is an interactive icon, it provides at least some feedback. If more is needed, then you can either attach a text icon to any interactive element (although this might become obtrusive and clutter the screen) or, for example, use different icons for interactive elements.

For example, the info icon is more likely to be considered an interactive icon by the user

than the room icon or the flag icon (as used in the example as it best fitted the idea of a bamboo pole marker).

Another issue to consider is the use on devices that don’t have cursor navigation, such as TV remote controls.

To support these, it is advisable to provide keyboard shortcuts as well.

For this, we first select the “Bamboo Text” image marker again, got to the “Settings” box and copy the Uuid. Then we select the “Bamboo Pole” icon marker again, go to “Settings” and go to the “Hotkey:” setting. Here we select an appropriate key (‘1’, for example) and type “toggleVisibility” (sorry, no pull-down menu here), followed by a colon and the Uuid into the field below.

This allows the user to display the info text about the bamboo pole by pressing the ‘1’ key. (It doesn’t change anything else – mouse hover and mouse click still have the same functions as before).

We then do the same in the “Bamboo Text” settings, so the user can remove it from view



page 25

again.

All marker related events are only active when the marker is visible, so pressing the ‘1’ key will only show the explanatory text image when the marker for the bamboo pole is shown. For the same reason, it is not sufficient to add the “Hotkey:” only to the image icon itself. While it is useful to turn the image off again, it can’t be used to turn it on.

It is also the reason why the image should have its own function (click and keypress) to turn it off again. Clicking on the flag marker (or pressing the key) only works while that is visible in the scene.

3.3.5 Intermezzo: Coordinates relative to scene or screen

When information is displayed on the screen, especially when it is text information, it is sometimes irritating that the information moves with the scene. In our example, the flag pole marker moves widely across the user’s field of view. If the explanatory text image is located at a fixed point in the scene (for example, always in the ‘forward’ view), it can happen that the user is looking at the flag pole marker as it vanishes in the distance behind. And when the description is activated, it will suddenly appear ‘behind’ the user and the user will not be aware of it and assume something didn’t work.

On the other hand, if the descriptive text moves with the flag pole marker, so it is always next to it, when it becomes visible, the text is moving in relation to the viewer, making it hard to follow and to read.

To avoid this, go to “Settings” for “Bamboo Text” and unselect Space and select Scene instead.

This will cause the image marker to be positioned in relation to the user viewing frame and not be handled as part of the 360° video space.

Then place the image at a suitable position in the frame.

As a result, the image will always appear at this position in the frame, regardless of what the user views in the video.



page 26

3.3.6 Tutorial 3: Controlling the view

The previous tutorials covered placing objects on the video and interacting with these objects. The control of the view in relation to the 360° video was left to the user.

NLV 360 also allows changing the viewport relative to the current view, to absolute positions and to follow marker positions. This tutorial covers all these options.

The first step is creating a “Look Forward” graphic and putting that on an accessible server somewhere.

As we want to have the button available from the start of the video, we jump to the start of the video in the editor. Now we move an “nlv-image” marker on the screen and place it on the lower left of the screen.

We now select the marker in the “Marker” box and go to the “Setting” box.

Here we give the marker a recognizable name, such as “Forward Button”. We also paste the URL of our graphic into the “Src:” text field. Then we uncheck “Space” and check “Scene” to ensure that the marker stays in the same screen position. Finally we select “navigateTo” from the “OnClick:” pulldown menu and enter “0;0” in the text field below.

For changes of the view, two parameters are required, the elevation and the direction, both given in degrees and separated by a colon. “0;0” denotes looking straight ahead in the video at horizon level. Entering “30,90” would cause the look to shift 30° up from the horizon and to a 90° angle to the left of the forward direction, “-45,-90” would look halfway down and to the right.

To add key press functionality, we also add a “Hotkey:” shortcut by selecting one of the keys (we’re using ‘9’ here) and entering ”navigateTo,0,0” in the text field below.

Commands and parameters are separated with a colon character, so this denotes the command “navigateTo” followed by two parameters, in this case two zeros. If we would, for example, pause the video as well, we would write ”navigateTo,0,0,pause”. Spaces should be avoided.

Finally, to complete the functionality of the button, we select “tooltip” from the “OnHover:”



page 27

pulldown and provide a short text to inform users who don’t understand the icon.

We now have an on-screen button that allows the user to reset the view with one mouse click or key press.

The button will change the view to a fixed position, in this case to forward view at horizon level.

It is also possible to change the view relative to the current view.

For this, we draw two buttons for left and right rotation with any graphics editing tool.



page 28

The next couple of steps are essentially the same as for the “Look forward” icon. We upload the images to a server, got to the beginning of the video, add “nlv-image” markers, give them useful names, paste the image URLs into the “Src:” field, uncheck “Space” and check “Scene”, place the images at a suitable place on the screen and provide a tooltip text for “On Hover:”.

But instead of using “navigateTo“ for the “OnClick:”function, we use “navigateRelativeTo”, with the parametes “0,30” in the entryfield below for rotating left and “0,-30” for rotating right. And, to allow for key navigation as well, we select a hotkey of ‘4’ for left and ‘6’ for right and enter ”navigateRelativeTo,0,30“ (or “navigateRelativeTo,0,-30”for right turns) in the textbox below.

We can now turn left and right and face forward again by using only the three onscreen buttons.

3.3.7 Intermezzo: Following objects

So far, we have navigated only relative to the camera orientation, either to absolute values (forward) or relative to the current view.

In the next step, we make our view follow an object in the scene.

At about 4:44 in the video, a dog sled overtakes the sled o which the camera is mounted, making the lead dogs of the overtaking sleds move from behind the sled to the front of the sled.

While the user can, of course, try to follow the action by adjusting the view as the dog passes, we will provide a button that follows the action automatically.

Again, we create a nice looking button, and upload it to the server.

Then we go through the usual step of creating an image marker with this image and anchoring



page 29

it on a fixed screen position. So we drag an “nlv-image” marker, on screen, give it a meaningful name (like “Overtaking”), put the URLs of the image into the “Src:” field, uncheck “Space” and check “Scene” , adjust the position on screen and provide a tooltip text (like “Watch overtaking”) for “On Hover:”).

So far the button is not doing anything.

For this, we use a new marker type – the “nlv-position”.

Essentially it is the same as an invisible “nlv-icon” marker. It can be put on the video and you can position it in keyframes, but it won’t be visible in the final video.

In this tutorial, we use it to mark the position of the lead dog of the overtaking team. So we select the “nlv-position” from the “Models” box and move it over the lead dog. We then watch the video and adjust the marker if the dog moves too far away, keeping the marker (more or less) on the dog during the video.

It is also advisable to go to the “Settings” box and give the marker some useful name, like “Dog Position”

The marker itself is quite small. In the image below, it is the little orange dot on the forehead of the dog.

.

Once we have the dog position marked throughout the video, we go to the “Settings” box for the marker and copy its Uuid.

We then select the image marker for the rightmost icon, go to the “OnClick:” field, select “autoFollow” from the pull-down menu and copy the Uuid of the marker to follow in the text field below. (And, for the sake of completeness, we also go to the “HotKey:” field, select a suitable key (“5”) and enter “autoFollow”, followed by a colon and the Uuid in the field below, to support key navigation as well.)



page 30

When we now click on that button, the view will swivel toward the position marker and keep following it, as long as the user does not interact with the view. So while the other sled is overtaking, the view will be centred on the lead dog of the overtaking sled.

If the user changes the view, either by clicking one of the other buttons or by dragging the view with the mouse, the autoFollow will stop. Clicking on the rightmost icon again will follow the dog again.

3.3.8 Tutorial 4: Navigating the timeline and chaining commands

In the previous tutorial, we created a button to allow the user to follow an action (following the lead dog of an overtaking sled). But as the action happens later in the video, the current user experience is confusing. If the user clicks the button early, the view will just swivel to the back (as the other sled is behind at that point) and stay there for almost five minutes before the overtaking takes place.

To avoid this, we want to add a button that jumps right to the moment the action happens.

We can go to a specific time in the video with the ‘seek’ function.

It would be simple to use the button, combine it with ‘seek’ and jump to the right moment in the video.

However, this would create another inconvenience for the user.

If the user actually watches the overtaking, changes the view (for example, to look ahead for a moment) and then presses the “Watch Overtaking” button again, the expectation is that the view will turn back to the lead dog. But what happens is that the video jumps back to the start of the scene, which is disruptive.

So we design the scene with two buttons, one that jumps to the scene and one that just follows.

Start End Button Function



page 31

0:00 4:44

Jump to 4:44, view follows

lead dog

4:44 5:54

View follows lead dog

5:54 End of Video

Jump to 4:44, view follows

lead dog

The easiest way to create the second button is to clone the first one.

For this we select the original “Overtake” button and create a clone with the duplicate button of

the “Markers” box, which looks like this:

We then select the new button, go to its settings and give it a different name. We also go to the “Src:” field and replace the previous icon with the new icon.

All the rest of the functionality is already there, except for the seek function itself.

So we go to the text field below “OnClick” and add “,seek,284000” after the uuid there and do the same for the text field for the “HotKey:”.

As commands and parameters are separated by colons, we have now added the “seek” command and the “284000” parameter to the click and key press functions. (The seek position is given in milliseconds, so 4:44 in the video is 284 seconds into the video or 284000 milliseconds.)

Both buttons are at the same position. We now have to make sure that only one of the buttons is visible at any time. For this, we go to the frames where we want the buttons to switch – at 4:44 and 5:54 in the video and move both buttons slightly (one pixel is sufficient) to create a movement keyframe here. We then just alternate the visibility of both markers, creating the effect that one is replaced by the other during the run of the video.



page 32

3.3.9 Tutorial 5: Overlays

In the second tutorial, we provided some additional information (about bamboo poles) by using an icon marker to trigger an image containing descriptive text.

While this gives the most control about the appearance of additional information, there is a shortcut for providing information – the overlay function.

It is, essentially, a full screen info display consisting of a text and an image, for which the interaction functionality is automatically provided by the NLV 360 system.

Let’s assume we want to provide an info-dump about dogsledding.

We select an ‘nlv-icon’, position it on the screen (as we want to provide information, we keep the default info icon) and go through the usual steps of naming it, scaling it, anchoring it on the screen position and providing a tooltip text on hovering.

Then we check the “Overlay” checkbox and fill the overlay related fields with the information we want to provide to the user – title, descriptive text, maybe a picture and an image title.

The “OverlayText” supports a limited number of HTML commands, most notably <br /> to force line breaks.



page 33

Once the overlay info has been entered, we select “overlay” from the “OnClick:” pulldown menu. And, for completeness, select a “HotKey:”, enter “overlay” in the text field below (to allow key activation of the overlay) and also provide an “OnHover:” tooltip text.

Activating the overlay, either by clicking on the icon or pressing the corresponding key, will pause the video and open an overlay that covers most of the view.

Overlays are best used in applications where the 360° scene works primarily as a selection view with multiple items placed around the camera, for example a camera placed between exhibits in a museum or on a conference table surrounded by people. Overlays can then provide details about the individual items shown. They are also useful for providing general information about the video, for example, listing actors, copyrights, background information and technical infos.

Since overlays disrupt the viewing experience, as they cover most of the screen and pause the video, overlays should only be sparingly used in narrative contexts.

3.4 Functional Description

The Interactive-Enriched Video Creation Tool IEVCT is used in the post processing phase of the PRODUCER workflow. IEVCT provides and Web based authoring tool (Editor) for creating the interactive elements for a flat or 360° video. It is also possible to link multiple videos together to create a story. IEVCT provides players for multiple platforms and considering different input capabilities like touch, motion, mouse, keyboard and TV remote control. The player can be easily integrated in application in a similar way to popular video services.



page 34

Furthermore, the IEVCT provides a set of APIs which allows the integration with other tools.

3.5 Mapping of tool’s functionality to the user requirements identified

Req ID Requirement Name (Description) Requirement mapping to tool’s functionality

C.9b

Reformatting the editing in several duration versions

Provides interactive annotation and interaction facilities and allows iterative building on earlier versions

B.6,B.9 Interactive Enriched Video Creation Tool Requirements

Provides annotation and interaction features to enrich bounded and 360° videos.

Table x. Tool requirements as identified in D2.1

3.6 Advantages and added value to the PRODUCER Platform

The tool allows interactive content to be easily added to existing video, supporting traditional video as well as 360° video. Especially documentary productions profit from being able to provide additional information to the video content presented, especially being able to provide supplementary content to increase understanding on the topics covered in the documentaries and highlighting interconnected issues and themes that are difficult to address in a linear medium as video. In addition, documentaries are likely to have a large amount of video content (in comparison to, for example, scripted productions) that has been created during the production process. An annotation tool allows creators to provide access to such supplementary material.



page 35

3.7 Architectural Diagram and placement in the overall architecture

Position of interactive annotation tool in overall architecture

Interactive annotation tool architectural diagram



page 36

3.8 Key Technologies

- Web based editor (HTML5, JavaScript, CSS3) - Node.js Backend - DynamoDB Database - RESTfull APIs - Playout for different device classes

- Key based navigation - Pointer based navigation

- Support for 360° content - Playout to Smart TVs

3.9 Code structure and availability

The tool runs on its own server as a dedicated service. APIs are provided via a JSON interface to allow export and import of database information about interactive content. The code is available at the GitLab internal git repository http://gitlab.fokus.fraunhofer.de/nlv/nlv-editor

3.10 Graphical Interfaces

The tool and its usage has been described in detail at the beginning of the chapter. The overall look of the tool as it appears in a browser is presented below.

http://gitlab.fokus.fraunhofer.de/nlv/nlv-editor



page 37

Interface of interactive annotation tool.

3.11 How to access the tool

The tool is accessible via: http://193.174.152.250:3004/. After login, follow the tutorial described above. Please note that the link refers to the testing instance and will be replaced in a later stage with the production URL.

3.12 Sample Use Cases (Workflows)

A typical workflow for a (fictional) documentation on work sledding has been presented in detail at the beginning of this section. In general, as a post-production tool, the annotation will be added after the primary documentary video has been finished.

The producer will then use the video and, typically, supplementary video, image and text materials to create an enhanced interactive version of the video.

On publishing (in the sense of making it publically accessible, distinguishing from a intermediate version, work version or unreleased version) the tool provides a URL to the interactive video player, which can then be referenced for web browser or SmartTV use.

3.13 Datasets used for tool testing

For tool testing and as a tutorial, a short 360° video of a dogsledding team has been used and enhanced with custom created content. Additional tool testing has been performed internally with a wide set of proprietary video content from various broadcasters (including PRODUCER partner Mediaset) to ensure that the tool is capable of handling the various video and distribution formats used by European broadcasters.

4 360° Video Playout


The 360° Video Playout enables the playback of 360° videos on a wide range of client devices especially HbbTV-enabled terminals for seamless integration with broadcast and broadband TV services. It enables high quality streaming of 360° video content without the need to increase the bandwidth compared to traditional videos. The 360° Video Playout addresses also low performance devices like TVs and Set-Top-Boxes that do not have the capability to process 360° videos locally. It offers players for a wide range of playback devices and platforms like HbbTV, Desktop, Android TV, Amazon FireTV, Chromecast and most Smart TV Platforms.


http://193.174.152.250:3004/



page 38


360 REQ1

360° analytics and object annotation well integrated

The 360° offers a player that can be fully integrated with the IEVT and provides 360° analytics like heatmap

360 REQ2

Availability of whole rendering and playout process

The 360° video rendering is accessible via a REST API and can easily be integrated with other tools

360 REQ3

Get Player URLs for multiple Platforms like HbbTV, Desktop, Android TV, etc.

There are players for all major TV platforms especially HbbTV available

360 REQ4

Video processing should keep enrichments along with the respective hyperlinks.

The 360° rendering operates on the videos and keeps the interactive elements defined in the Video

Tool requirements as identified in D2.1


Documentaries are considered as the best suitable genre for 360° videos and many broadcasters are looking for production of 360° documentaries. Head-mounted displays (HMDs) are today's the most popular devices for watching 360° content but the vision of the 360° Video Playout is to enable a novel interactive, clickable 360° video experience for all platforms but especially for TV devices. It gives the viewer the opportunity to interact with 360° video content on the TV just by using the TV remote control. Thereby it leverages the viewer’s personal involvement, while content provider and broadcaster may benefit from getting (anonymous or personalized) viewer statistics such as interaction behaviour, the viewing path or the most frequently visited scenes. Such a combination of interactive 360° video and analytics leads to value-added information for content creators to create content smarter and to break new ground of storytelling.

4.4 Architectural Diagram and Placement in the overall architecture

The 360° Video Playout for HbbTV consists of the following components:

● 360° Video Pre-processing: The Cloud Rendering Server operates on the source 360° video and calculates the different FOVs depending on the configuration that is also passed as input. A FOV is defined using the four parameters (φ, θ, Ah, Av). φ, θ define the center of the FOV angle across the horizontal space (φ) and vertical space (θ). Ah, Av define the horizontal and vertical dimensions of the FOV in degree. Typical values are Ah= 105° and Av=60°.



page 39

● Storage and Streaming: The Cloud Streaming Server offers a REST API to control the individual view and playback state for each client. The output FOVs and manifest of pre-processing steps need to be available on a streaming server that is accessible from the CDN provider.

● 360° Video Player Client: The Cloud Streaming solution enables Smart TVs and other streaming devices to provide 360° Video experience through usual video playback of the output from the cloud streaming server. Users can control their individual view and playback state via remote control. The player constructs the final video stream to display from the FOV videos. There is no need to process the received video content before playback. The client platform needs only to provide an API that allows applications to send video segments to the video decoder.

● 360° Video Control Mobile App: The mobile control app acts as a smart remote control for the 360° Video TV Player. It offers an easy-to-use UI to control the video playback and the angle of current view using touch, gestures or device orientation.

The following diagram shows the interworking between the components described above on HbbTV. The 360° video is served as input to the Cloud Renderer which renders a FOV and sends it as live stream (TS) to the HbbTV terminals. The viewer can change the FOV using a second screen application which sends the control commands to the REST API of the Cloud Renderer or by using the TV remote control. In this case, the HbbTV application running on the TV intercepts the remote control commands and send them to the same REST API of the Cloud Renderer used by the second screen application.

For new HbbTV terminals that support the W3C Media Source Extension API (MSE) there is better architecture depicted in the following diagram which supports streaming via content delivery networks (CDNs). This solution will also apply to devices like FireTV, Android TV,

Chromecast and Desktop. The architecture comprises the four steps “Pre-processing”, “Storage”, “Streaming”, and “Playback”. During the pre-processing step, a source 360° video

will be pre-rendered with different FOV combination and the output FOVs will be stored somewhere in network storage. In the next step, the created FOVs will be analyzed and a



page 40

manifest file will be created. After the pre-processing is completed, all FOVs and the manifest file will need to be made available on a storage server that is accessible by a CDN provider. For streaming the content to the client, the CDN infrastructure of the provider will be used. In the last step (i.e. “playback”), the player reads and parses the manifest file to get information about available FOVs and how to access them. The player starts playback with the default

FOV and listens to requests from the input device to navigate to another FOV.


- HbbTV TS Streaming: The 360° Video Playout supports TS Streaming which is already available in all HbbTV Terminals since HbbTV 1.0. This method ensures compatibility with old devices but it is not the prefered streaming method since it requires 1-1 Connection between client and server which means that Content Delivery Networks cannot be used and on the other side it is not possible to control the buffer on the TV terminal which may produce some delay on some terminal.

- DASH packaging: For other non-HbbTV Web platforms and also for new HbbTV models that support the new W3C Media Source Extension API we use the DASH standard as streaming format and ISOBMFF as the file format in order to guarantee playback interoperability across devices.

- Web APIs (MSE, EME, XHR) for Player: On the client-side we leverage Web technologies such as W3C Media Source Extension API (MSE). MSE API allows Web applications to control the source buffer of a HTML5 video object by appending, removing or replacing segments. No Canvas API is needed since the 360° video is already pre-rendered. Therefore, the content can also be DRM-protected and played with the help of W3C Encrypted Media Extensions (EME). We use a single MSE SourceBuffer for seamless transitions between FOVs. Multiple SourceBuffers could cause video decoding interrupts. By pre-buffering FOVs adjacent to the



page 41

current FOV, the switching delay between FoVs can be optimized. Using MSE’s appendBuffer ISOBMFF segments of a FoV are fed into the SourceBuffer for playback. When the FoV changes, existing segments are replaced by segments of the new FoV. Moreover, the adaptation logic in a DASH player needs to be modified for this type of playback. Besides pre-buffering of adjacent FOVs, the different bitrate Representations can be used to further optimize FOV switching latency. For example, when the user is switching a lot between FOVs, only the lowest bitrate of the low-latency Representations is requested from the CDN. Once the FOV is unchanged and playback stabilizes, adaptation logic can decide to switch to higher bitrates. Furthermore, for requesting FOV video segments we use the new W3C fetch API instead of XHR API. The fetch API allows the client to access downloaded chunks before the whole content is downloaded. In this case the player can request multiple segments in a single request (e.g. using HTTP Range header) and still be able to access each segment as soon as all its chunks are downloaded.


The 360° Video Playout is available only in the Fraunhofer FOKUS internal git repositories and not published as open source:

● 360° Video Streaming and Playback: https://gitlab.fokus.fraunhofer.de/famium/famium-server

● 360° Video Manager and Pre-renderer: https://gitlab.fokus.fraunhofer.de/fame-wm/360_Video_Manager_and_Gateway/

4.7 Programming Interfaces

The 360° video playout offers as depicted in the architecture multiple components. The Pre-Rendering Server provides a REST API that can be used to prerender a video as described in the architecture above. The REST API accepts JSON requests with the following information in the body:

{

"videoUrl": "https://s3-eu-west-1.amazonaws.com/fame-video-store/xxxx.mp4",

"callbackUrl": "http://52.59.90.254:3001/my/callback",

"startPhi": 0,

"startTheta": -60,

"stepPhi": 30,

"stepTheta": 90,

"endPhi": 360,

"endTheta": 60,

"modes": ["static","motionl","motionr","motionu","motiond"],

"fov": 60,

"gopSize": 10

https://gitlab.fokus.fraunhofer.de/famium/famium-server


https://gitlab.fokus.fraunhofer.de/fame-wm/360_Video_Manager_and_Gateway/

https://gitlab.fokus.fraunhofer.de/fame-wm/360_Video_Manager_and_Gateway/



page 42

}

Below is a description of the parameters in the JSON body:

● videoUrl: The URL of the input 360° video ● callbackUrl: The URL on which the response will be sent when the prerendering is

completed ● startPhi: The start horizontal angle FOV angle ● startTheta: The start vertical angle FOV angle ● stepPhi: The step between two horizontal FOV angles ● stepTheta: The step between two vertical FOV angles ● endPhi: The end horizontal angle FOV angle ● endTheta: The end vertical angle FOV angle ● modes: An array of modes which must be a subset of the ["static" ,"motionl", "motionr",

"motionu", "motiond"]. “static” mode is mandatory. ● fov: The FOV viewport vertical angle (has impact on zoom level). ● gopSize: Number of frames in a GOP.

The JSON response of the prerendering sent to the callback URL is depicted in the following JSON example. It includes both player URLs for HbbTV and other browser environment.

{

"playerHbbTVUrl": "http://193.174.152.250:3000/hbb.html?uid=_d5g7q6k8y9hawfe",

"playerMSEUrl":"http://193.174.152.250:3000/mse/player.html?uid=_d5g7q6k8y9hawfe",

}


The simplest Workflow consists of the following steps:

Multimedia Content Storage

360° Video

Prerenderer

360° Video Streamer &

Controller

360° Video

Player

PRODUCER Management

Frontend

Upload

Trigger Request

Stream



page 43

● Step1: A new 360° video is uploaded in the Multimedia Content Storage. ● Step2: Send a request to the 360° Video Playout Prerendering API as described

above. Provide a callback URL in the request. ● Step3: When Prerendering is completed, a new HTTP POST request with location of

the new prerendered video folder and link to player URL will be sent to the callback URL.

● Step4: The Player URL received in previous step can be integrated in any HbbTV application or other applications like FireTV and Chromecast using the 360° Video player.


The current dataset used for testing is the Big Brother video provided by MediaSet. The video is available at http://193.174.152.250:3000/mse/player.html?uid=_d5g7q6k8y9hawfe&gopCorrect=-1&autoplay=1

5 Second Screen Framework


The Second Screen Framework (SSF) is an implementation of state-of-the-art standards and technologies related to key multiscreen features such as Messaging, Discovery, Application Launch, Content Presentation, Signalling, Synchronization, Push and Pairing of devices. SSF makes the development of multiscreen applications across different domains like Mobile and TV faster and easier. The core of the framework is based on web technologies and enables access to underlying components using standard interfaces like W3C Presentation API [25] and HbbTV 2.0 Companion Screen API [26]. Each SSF component provides implementations for state of the art protocols and technologies like SSDP, mDNS, Physical Web, Airplay, Miracast, DIAL and UPnP and offers Software libraries and tools that can be easily integrated in applications on different platforms like Android, iOS, Cordova, LG WebOS TV and Node.js. SSF provides also backend services that are accessible via REST interfaces and WebSockets.



SSF REQ1

Interaction with interactive AdWeb pages

The PRODUCER Apps can use the SSF to link Ad displayed on the TV and open an AdWeb Page on second screen using the Push feature of the

http://193.174.152.250:3000/mse/player.html?uid=_d5g7q6k8y9hawfe&gopCorrect=-1&autoplay=1

http://193.174.152.250:3000/mse/player.html?uid=_d5g7q6k8y9hawfe&gopCorrect=-1&autoplay=1



page 44

SSF

SSF REQ2

Main and Second Screen Applications run in a Browser Context for example HbbTV on TV sets, Browser on Desktop and Mobile Browser or Hybrid Applications on mobile platforms like Android and iOS

The SSF supports all Web Platforms even old Browsers and HbbTV devices. It provides a fallback mechanisms to support all different browsers.

SSF REQ3

Non-video content for second screen consumption

The App Launch and Messaging features of the SSF can be used to open a second screen app on mobile that interact with the video on the TV and also for synchronizing content.

SSF REQ4

(Pseudonymous-) Linking of TV to specific user device (could be ID number, QR code or DLNA discovery, etc.

QR-Code and PIN code are supported. DLNA discovery is supported only in new HbbTV models using and the second screen app must be a hybrid application.

SSF REQ5

Automatic device discovery, pairing and synchronization

SSF provides a mechanism to automatically connect devices which are paired before.

SSF requirements as identified in D2.1


SSF facilitates the discovery, pairing, launch, communication and synchronization between Smart TV and second screen devices. It can be used form example for enabling the delivery of Interactive TV Ads, which trigger the access to extended contents and e-commerce feature directly from the TV Ad. It can be also used to synchronize multimedia content between TV and Second Screen for example to provide personalized audio tracks for the same documentary. SSF can also be used to facilitate interactive videos especially interactive 360° documentary videos where the user get more insights about what the content he/she is watching on the TV and can interact with the content or request additional data. Furthermore, in a 360° video, the user can get information about Regions of Interest (ROIs) which are not visible in the current Field of View. This is one of the most important/significant added values second screen brings to 360° videos running on the TV.



page 45

5.4 Architectural Diagram and Placement in the overall architecture

The SSF consists of three layers, as depicted in the Architecture diagram above.

1. The first layer is the Standard Interfaces layer that allows developers to get access to the underlying multiscreen screen features via standard APIs. SSF supports the W3C Presentation API and the HbbTV 2.0 Companion Screen API in addition to a set of custom REST APIs.

2. The second layer is the Libraries and Tools layer that provides an implementation of each of the multiscreen features. These libraries expose a set of functions to the first layer. The main multiscreen features are:

a. Discovery: allows an application e.g. on Smartphone to discover suitable devices in a Home Network e.g. TVs.

b. Launch: allows to launch an application on a discovered device remotely e.g. a smartphone app can launch an HbbTV App.

c. Messaging: allows two applications to exchange messages between each other in both directions.

d. Synchronization: allows to synchronize media streams and other content across multiple devices.

e. Signaling: is the channel for exchanging signaling information between different applications e.g. when a device becomes available or unavailable or when a messaging channel gets connected or disconnected.

f. Pairing: allows to pair devices without discovery for example by using a PIN code and scanning a QR-code. After pairing, the devices can connect to each other until they are unpaired.

g. Push: allows to launch applications on smartphones and tablets. This is only possible via push notification which is the best option from a usability perspective.

3. The third layer is the Protocols layer which provides implementations of the state-of-the-art protocols that are used in the implementation of the components of the second



page 46

layer. Many of the protocols are supported in all modern browsers like HTTP or WebSocket. Other protocols like SSDP, mDNS, DIAL can be implemented on top.


The SSF implements the W3C Presentation API introduced by the W3C Second Screen Community Group and continued on the W3C Second Screen Working Group. SSF implements also the Companion Screen and Media Synchronization Protocols introduced in the latest version of the HbbTV 2 Standard.

W3C Presentation API: The main purpose of the Presentation API is to enable web pages to access external presentation-type displays and use them for presenting web content. It aims to make secondary displays such as a projector or a connected TV available to the web and takes into account displays that are attached using wired (HDMI, DVI or similar) and wireless technologies (MiraCast, Chromecast, DLNA, AirPlay or similar). At its core, the specification of the Presentation API enables an exchange of messages between a requesting page and a presentation page shown in the secondary display. How those messages are transmitted is left to the User Agent (UA) in order to allow for use of display devices that can be attached in a wide variety of ways. For example, when a display device is attached using HDMI or MiraCast, the UA on the requesting device can render the requested presentation page in that same UA, but instead of displaying in a window on that same device, it can use whatever means the operating system provides for using those external displays. In that case, both the requesting page and the presentation page run on the requesting device and the operating system is used to route the presentation display output to the other display device. The second display device doesn't need to know anything about this spec or that the content involves HTML5. Alternately, some types of external displays may be able to render HTML5 themselves and may have defined their own way to send messages to that content. In that case, the UA on the requesting device would not need to render the presentation page itself. Instead, the UA could act as a proxy translating the request to show a page and the messages into the form understood by the display device.

HbbTV 2: The HbbTV 2 specification, published in 2015 by the Hybrid Broadcast Broadband TV (HbbTV) Association, introduces a number of new features that support companion screens and improve the synchronization of broadcast and broadband content. The SSF implements these features offering a set of modules and libraries that can be easily integrated in existing TV platforms or broadcaster's services to make them more attractive for end-users. HbbTV 2 CS and Synchronization offers the following features:

1. Discovery of HbbTV Terminals and Companion Screens: allows an HbbTV application to list available companion screens and a companion application to discover HbbTV terminals in the same network using SSDP.

2. Launching an HbbTV application from a CS application: allows a CS application to launch an HbbTV application on the TV using DIAL protocol.

3. Launching a CS application from an HbbTV application: allows an HbbTV application to launch a CS application on second screen. The FOKUS framework offers a CS Launcher for multiple mobile platforms and supports Push Notification via Google's GCM or Apple's



page 47

APN. 4. Application to Application Communication: allows an HbbTV and CS applications to

exchange data via an App2App Communication Server that supports WebSockets. 5. Multi-Stream Synchronization: allows an HbbTV application to synchronize multiple

media streams. It supports Broadcast-Broadband and Broadband-Broadband synchronization.

6. Inter-Device Synchronization: allows to synchronize Broadcast or Broadband streams on TV with broadband content on companion screen.


Some components of the SSF are available as open source on github and the javascript package manager npm. The following list provides links to all repositories.

● HbbTV 2.0 CS Node.js Module: ○ github: https://github.com/fraunhoferfokus/node-hbbtv ○ npm: https://www.npmjs.com/package/hbbtv

● HbbTV 2.0 CS Cordova Plugin: ○ github: https://github.com/fraunhoferfokus/cordova-plugin-hbbtv ○ npm: https://www.npmjs.com/package/cordova-plugin-hbbtv

● HbbTV 2.0 CS Cordova Mobile Test App: ○ github: https://github.com/fraunhoferfokus/cordova-plugin-hbbtv-helloapp

● DIAL Node.js Module: ○ github: https://github.com/fraunhoferfokus/peer-dial ○ npm: https://www.npmjs.com/package/peer-dial

● SSDP Node.js Module: ○ github: https://github.com/fraunhoferfokus/peer-ssdp ○ npm: https://www.npmjs.com/package/peer-ssdp

● Presentation API Cordova Plugin: ○ github: https://github.com/fraunhoferfokus/cordova-plugin-presentation

Furthermore, the main SSF project which provides a pure JavaScript implementation and is compatible with all modern and old browsers on all platforms including old HbbTV devices is available only in the Fraunhofer FOKUS internal git repository and not published as open source: https://gitlab.fokus.fraunhofer.de/famium/famium-server

5.7 Programming Interfaces

SSF is a framework and doesn’t offer a user interface, but instead it offers a set of libraries that can be integrated in the PRODUCER end user applications. Since PRODUCER is mainly addressing web technologies and HbbTV, only the Web libraries are relevant for it. Below is a description on how to access framework features via the JavaScript APIs.

5.8 How to access the framework

The library name for SSF is called FAMIUM. It implements the W3C Presentation API with additional JavaScript APIs that are not part of the Presentation API.

https://github.com/fraunhoferfokus/node-hbbtv

https://www.npmjs.com/package/hbbtv

https://github.com/fraunhoferfokus/cordova-plugin-hbbtv

https://github.com/fraunhoferfokus/cordova-plugin-hbbtv

https://www.npmjs.com/package/cordova-plugin-hbbtv

https://github.com/fraunhoferfokus/cordova-plugin-hbbtv-helloapp

https://github.com/fraunhoferfokus/peer-dial

https://github.com/fraunhoferfokus/peer-dial

https://www.npmjs.com/package/peer-dial

https://github.com/fraunhoferfokus/peer-ssdp

https://github.com/fraunhoferfokus/peer-ssdp

https://www.npmjs.com/package/peer-ssdp

https://github.com/fraunhoferfokus/cordova-plugin-presentation


page 48

The following JavaScript Library (webcreens.all.min.js) that implements the Presentation API needs to be included in the web pages that use the API:

<script src="http://famium.fokus.fraunhofer.de/js/webscreens.all.min.js"></script>

The following HTML examples sender.html and receiver.html illustrate the usage of the API. sender.html is the web page for the controlling device (second screen) and receiver.html is the web page to be displayed on the TV or already launched via HbbTV.



<!doctype html>

<html>

<head>

<title>Sender Example</title>

<script src="http://famium.fokus.fraunhofer.de/js/webscreens.all.min.js">

</script>

<script type="text/javascript">

navigator.presentation.onavailablechange = function(dict){

if(dict.available){

var session = navigator.presentation.requestSession("receiver.html");

session.onmessage = function(msg){

console.log("onmessage",msg);

};

session.onstatechange = function(){

console.log("onstatechange",session.state);

if(session.state == "connected"){

session.postMessage("foo");

}

}

}

};

</script>

</head>

<body>

<h1>Sender Example</h1>

</body>

</html>

page 49

receiver.html includes also the FAMIUM JavaScript library webscreens.all.min.js and uses navigator.presentation.onpresent to receive session requests from sender.html. The session can be used in the same way as for sender.html.



<!doctype html>

<html>

<head>

<title>Receiver Example</title>

<script src="http://famium.fokus.fraunhofer.de/js/webscreens.all.min.js">

</script>

<script type="text/javascript">

navigator.presentation.onpresent = function(evt){

var session = evt.session;

session.onmessage = function(msg){

console.log("onmessage",msg);

session.postMessage("bar");

}

session.onstatechange = function(){

if(session.state == "disconnected"){

window.close();

}

}

};

</script>

</head>

<body>

<h1>Receiver Example</h1>

</body>

</html>

To run the Example application, both HTML files sender.html and receiver.html need to be hosted on a web server (e.g. http://example.com) in the same folder (public URLs could be for example http://example.com/sender.html and http://example.com/receiver.html). In a next step one or more Virtual Displays need to be opened in a Browser on any device using the following URL: http://famium.fokus.fraunhofer.de/display.html#room=[room]&name=[name]. [room] and [name] need to be replaced with the actual values e.g.



page 50

http://famium.fokus.fraunhofer.de/display.html#room=12345678&name=MyDisplay. The parameter room assigns one or more virtual displays into a group. The parameter name specifies a friendly name of the Display. A display in a room can be found only by applications in the same room. To assign the example application above to a room, the sender application need to be opened in a web browser using the following URL: http://example.com/sender.html#room=[room]. [room] needs to be replaced with the actual room value e.g. http://example.com/sender.html#room=12345678. In this case the example application will find the virtual display MyDisplay since both are in the same room.

The implementation of the Presentation API webscreens.min.js connects automatically to the server when the hash part of the URL contains at least room parameter (e.g. http://example.com/sender.html#room=abc). In this case if only the room parameter is provided, the page will be considered as sender. If the hash part of the URL contains room and name parameters (e.g. http://example.com/receiver.html#room=abc&name=FooPresentation), then the Presentation API connects also automatically to the server but the page is considered as receiver. This is what happens under the hood when you launch a receiver page e.g. http://example.com/receiver.html on a display e.g. http://famiunServerHost/display.html#room=abc&name=MyDisplay. The display adds automatically its hash parameters (in our example #room=abc&name=MyDisplay) to the receiver URL (final receiver URL in our example will be then http://example.com/receiver.html#room=abc&name=MyDisplay) and opens the final receiver URL in an <iframe> in fullscreen. The Presentation API lib webscreens.min.js of the receiver page connects then automatically to the server because both parameters room and name are provided in the hash part of the URL. After the receiver is launched successfully, the selection dialog of the sender page will offer two options in the selection dialog both with the same name (in our example name=MyDisplay): Option 1) is to launch the receiver page on the display which will close and override the old receiver page and option 2) is to join the current receiver without launching a new page. Option 2) is relevant for multi-user applications where two or more senders need to connect to the same receiver (We will use also the term 'presentation' for an already launched receiver page). This means also that you can start receiver pages with room and name parameters manually in the Browser without using any display and the sender will only offer the join option. To make this clearer, try the following example:

1. start receiver: open http://famium.fokus.fraunhofer.de/apps/helloworld/receiver.html#room=abc&name=HelloWorldPresentation in a browser tab.

2. start sender: open http://famium.fokus.fraunhofer.de/apps/helloworld/sender.html#room=abc in a browser tab on same or any other device and click on 'select display'. In the dialog you will get the option Join Presentation 'HelloWorldPresentation'?. When you click on OK, you will join the presentation started in previous step.

3. start display: open



page 51

http://famium.fokus.fraunhofer.de/display.html#room=abc&name=MyDisplay in a browser tab.

4. back to sender: go to sender page started in step 2 and click on 'select display' again. Now you will get two options 1) to select display 'MyDisplay' to start a new presentation or 2) to select presentation 'HelloWorldPresentation' to join it. Select option 1) to start a new Presentation on display 'MyDisplay'. Now you will have two running presentations 'HelloWorldPresentation' and 'MyDisplay' and one display 'MyDisplay'. This is why if you click on 'select display' again you will get three options.

In many cases, the sender page needs to offer a selection dialog (or any way to select a display or presentation) that is inline with the design of the application. The only reason why we use the selection dialog offered by FAMIUM is to implement the Presentation API according to the spec without touching the DOM tree. To disable this default behaviour, FAMIUM offers a set of additional APIs that can be used on the sender page to monitor the availability of displays and presentations in a specific room and to launch or join a presentation using its ID or the ID of the display. These APIs are listed below:

● famium.webscreens.onadddisplay: The onadddisplay event listener will be triggered each time a new display is available. The new display will be passed as input. The display input parameter is a JSON object that contains two properties id and name. The application can use this function as trigger to update the display list in the UI by adding the new display.

famium.webscreens.onadddisplay = function(display){

console.log("new display available",display);

console.log("- display.id",display.id);

console.log("- display.name",display.name);

// TODO: update the display list in your UI

}

● famium.webscreens.onremovedisplay: The onremovedisplay event listener will be triggered each time a display disappears. The disappeared display will be passed as input. The display input parameter is a JSON object that contains only one property id. The application can use this function as trigger to update the display list in the UI by removing the display.

famium.webscreens.onremovedisplay = function(display){

console.log("display disappeared",display);

console.log("- display.id",display.id);

// TODO: update the display list in your UI



page 52

}

● famium.webscreens.onaddpresentation: The onaddpresentation event listener will be triggered each time a new presentation (receiver page) is launched. The new presentation will be passed as input. The presentation input parameter is a JSON object that contains the properties id, name and url. The application can use this function as trigger to update the presentation list in the UI by adding the new presentation.

famium.webscreens.onaddpresentation = function(presentation){

console.log("new presentation available",presentation);

console.log("- presentation.id",presentation.id);

console.log("- presentation.name",presentation.name);

console.log("- presentation.url",presentation.url);

// TODO: update the presentation list in your UI

}

● famium.webscreens.onremovepresentation: The onremovepresentation event listener will be triggered each time a presentation is closed. The closed presentation will be passed as input. The presentation input parameter is a JSON object that contains only one property id. The application can use this function as trigger to update the presentation list in the UI by removing the presentation passed as input.

famium.webscreens.onremovepresentation = function(presentation){

console.log("presentation closed",presentation);

console.log("- presentation.id",presentation.id);

// TODO: update the presentation list in your UI

}

● famium.webscreens.displays: The list of available displays will be maintained in famium.webscreens.displays so there is no need to maintain the displays in your application. famium.webscreens.displays is a JSON object where the keys are the IDs of the displays and the values are the same display JSON objects passed as input to famium.webscreens.onadddisplay. PS: when your sender application is loaded, you need first to check famium.webscreens.displays because famium.webscreens.onadddisplay considers only new added displays that are started after your sender application.

● famium.webscreens.presentations: The list of available presentations will be



page 53

maintained in famium.webscreens.presentations so there is no need to maintain the presentations in your application. famium.webscreens.presentations is a JSON object where the keys are the IDs of the presentations and the values are the same presentation JSON objects passed as input to famium.webscreens.onaddpresentation. PS: when your sender application is loaded, you need first to check famium.webscreens.presentations because famium.webscreens.onaddpresentation considers only new launched presentations that are started after your sender application.

● famium.webscreens.connect(): As mentioned before, FAMIUM connects automatically to the server if room parameter is available in the hash. If this is not desired for your application logic or you desire to pass the room parameter in a different way as by using room hash parameter, in this case you can skip the room parameter from the hash and use famium.webscreens.connect() to connect manually. famium.webscreens.connect() returns true if successful or false if it was already connected because room parameter was provided in the hash or famium.webscreens.connect() was already called before. IMPORTANT: receiver pages launched on a display will always have a room and name hash parameters as explained in the beginning of this section this means receiver pages launched in that way will automatically connect to the server. In this case if famium.webscreens.connect() is used in the receiver page it will always returns false and will not have any effect. If you start a presentation manually and you don't use room and name hash parameters, you can use famium.webscreens.connect() to connect manually. It is recommended to use the default behaviour in receiver pages since manual connect and join rooms will not bring any advantage comparing to the default behaviour.

● famium.webscreens.join(room[,name]): If famium.webscreens.connect() is used to manually connect to the server, famium.webscreens.join() can be used after famium.webscreens.connect() to join a room. famium.webscreens.join(room[,name]) accepts two parameters the first one is room and is mandatory and the second one is name and is optional. if only room is provided, the application will be considered as sender. If both room and name are provided, then the application will be considerd as receiver. A sender application can join multiple rooms but a receiver application only one room.

● famium.webscreens.leave(room): famium.webscreens.leave(room) can be used to leave the room passed as input.

● navigator.presentation.requestSession(url[,config]): navigator.presentation.requestSession(url) is specified in the Presentation API and accepts as input the URL of the receiver page/presentation. We extended this function and allow a second optional parameter config which is a JSON object that contains one of the keys presentatonId or displayId. The value of presentatonId must be a valid Presentation ID. The same for the value of displayId it must a valid display ID. If config is passed as input, then the default selection dialog will not be displayed and the receiver page will be launched on the display if displayId is passed. If presentatonId is



page 54

passed then the target presentation will be joined.


Let us consider a PRODUCER application for HbbTV which shows an interactive 360° Video on the TV. The viewer can use the TV remote control arrow keys to navigate in the 360° video or if interactive mode is selected, the viewer can select interactive elements and trigger them using the OK key. The PRODUCER HbbTV App offers also a second screen application which makes the interaction much simple than just using the TV remote control but also offers additional features which cannot supported by the TV application.

6 Social Recommendation and Personalization Tools


The Social Recommendation and Personalization tool consists of three main mechanisms:

1. Personalization 2. Relevance Feedback 3. Social Recommendation

which are combined in order to be able to provide meaningful recommendation to the users. The tool allows the search and filtering of content based on the user’s interests, improving his Quality of Experience and making the overall PRODUCER toolkit’s usage more appealing. The SRP tool is responsible for ranking the video content, as well as the set of enrichments accompanying the video in such a way that the user can easily find the content that interests him the most.

In the following subsections we will elaborate more about each of the mechanisms.

6.1.1 Personalization

The Personalization mechanism’s goal is to provide content that satisfies the individual needs . Each user is considered an entity that has personalized characteristics such as taste, age, viewing history etc., characteristics that are integral parts of the system. In literature, personalization is achieved by deducing the user’s needs and taste through one-to-one interactions with the user [1]. In order to make meaningful recommendations, accurate modeling of the interests of the users is important. Such representations are referred as User Model [2]. In order to collect the appropriate data to create an accurate model, two methods are available: 1) the explicit way where the user is supposed to explicitly state his\her interests and 2) the implicit way where the model is built using information collected from the observable behavior of the user [3]. The explicit way is supposed to be more accurate but is more difficult to collect since the user typically avoids any explicit feedback. On the other hand, the implicit way is more error-prone in misinterpreting the user’s behavior. In practice, the explicit and implicit approaches combined produce more accurate results [4].

The SRP tool models the users as nodes connected with term nodes that correspond to topics/genres of the content. Each edge connecting the user with the different term nodes has



page 55

a weight that signifies the correlation between the user and the term. The higher the value, the more the user is interested in that kind of content. Since the content is modelled in a similar way, it is possible to find the most appropriate content for the user. The weights of the edges between user nodes and term nodes is constantly updated based on the interactions and behaviour of the user within the tool.

6.1.2 Relevance Feedback

The Relevance Feedback mechanism’s goal is to constantly update the user’s model described earlier in order to better match the user’s interests. The importance of the mechanism lies not only on finding the correct user profile but being able to adapt it as the user changes his interests over time.

In the literature, there are a number of formal ways of describing relevance feedback and most of them are related to the notion of an “optimal user query”. The optimal query is defined as a vector obtained by taking the difference between the relevant and non-relevant sets of items that the user selects, also represented as vectors [5]. This relevance feedback approach can be modified to a weighted average of the identified relevant items and, in some versions of the algorithm, the identified non-relevant items. The revised approach is then used to produce a new item ranking [6]. Another common way of describing relevance feedback is implicitly using Bayesian classification models of retrieval [7]. In this approach, identified relevant items are used to estimate the characteristics of the relevant class of items for a given context. The characteristics are used to estimate probabilities for the non-relevant class. The revised estimates are used to produce a new item ranking based on the probability of belonging to the relevant class. Improving user experience by relevance feedback demonstrates good results by adding relevance information in a relevance feedback environment [8] Considering this framework, the overall effectiveness increases depending on the amount of relevant information that has been added to the relevance feedback mechanism.

On the previously mentioned approaches, the proposed mechanisms consider relevance feedback coming from the interaction with the documents themselves. Using relevance feedback from enrichments associated with the video, each one having its own representation in the modelling system we propose, has not been researched extensively. The SRP tool tries to be a concrete tool using user’s relevance feedback on that type of information as well.

The SRP tool makes use of explicit as well as implicit feedback provided by the user as the user watches a video. Concerning the explicit feedback, the user can state his\her interest in a particular video by clicking a “like” or “dislike” button and thus giving a “+1” or “-1” relevance score. By ignoring the explicit relevance feedback, an “indifferent” signal of “0” score is considered. The implicit feedback concerns all observable interactions of the user with the content. As a result, the implicit feedback includes the following:

● Selection of video ● Playtime of video ● Share video ● Click on enrichments ● Share enrichment



page 56

6.1.3 Social Recommendation

The Social Recommendation mechanism’s goal is to collect and analyse different sources of information in order to provide ranking and scores on the content tailored to each separate user.

In order to produce the recommendations, several steps have to be followed, such as collecting the right type of information concerning the users and the items, the availability of that kind of information, the correlation between the users and the items, the algorithmic techniques used to rank the content, the filtering of the results based on novelty, coverage and precision etc. [9]. Based on the information used to provide the recommendations, the recommender systems are classified in four different categories:

1. Content-based recommendations that derive from the comparison between the user’s past actions and the nature and characteristics of the items [10].

2. Demographic filtering that makes use of the personal information of the user such as age and gender to make predictions of the preferences he has on the content [11]. This makes the assumption that users will have similar tastes if they are members of the same demographics group.

3. Collaborative recommendations that make use of the information gathered from similar users to the user in question. One of the most commonly used algorithms used in collaborative filtering is the k nearest neighbors algorithm which applies a similarity metric to determine the k most similar users and provide recommendation based on them [12]. Several types of similarity metrics can be considered such as cosine, Jaccard, Euclidean etc. [13].

4. Hybrid recommendations that try to combine the above recommendation mechanisms in order to make use of the advantages offered by each one of them [14].

Recommendation systems also can make use of data coming from online social networks (e.g. Facebook, Twitter etc.) to enhance their performance by collecting information such as the demographics of users, posts, followers, tags etc. [15].

The Social Recommendation & Personalization tool makes use of a hybrid recommendation system by combining suggestions based on the individual user’s profile with that coming from people with similar taste. The philosophy behind that choice is that the content-based recommendation tends to recommend the same type of content to the user, making novelty difficult to achieve.

Apart from the suggestions to the viewer, the SRP tool is responsible to provide recommendations to the content providers. To do so, the tool makes use of the information it has already collected in its database. By filtering its users according to the target group provided, it uses clustering techniques (e.g. k-means [16]) to find a representation of the users’ interests and provides its recommendations accordingly.


Req ID Requirement name Requirement mapping to tool’s functionality



page 57

SRP REQ1

User profiling functionality The tool does create a user profile that is used to suggest appropriate content

SRP REQ2

User profiling: Query result to be ranked based on user's profile

The content is ranked based on the dynamically produced user’s profile

SRP REQ3

User profiling: Pause time of video (duration)

The pause time of the video is not used since there is no effective way to know why the video was paused and

what it means on the user’s preference

SRP REQ4

User profiling: Stop timestamp of video

The stop timestamp is used to calculate the duration of the video that was actually watched by the user. Longer watching times signify appreciation of the content by the

user.

SRP REQ5

User profiling: Click on enrichments

Clicking on an enrichment for more information shows interest of the user to the enrichment and the information

is used to update the user’s profile.

SRP REQ6

User profiling: Share of enrichment in social networks, e.g. FB, Twitter, e.t.c.

Sharing an enrichment is also perceived as interest of the user to the enrichment and the profile is updated

accordingly.

SRP REQ7

User profiling: Explicit relevance feedback related to video (maybe after the playtime)

Explicit user feedback is used since it directly shows the user’s liking of the content.

SRP REQ8

Find #k similar users to the target user and/or users' clusters for collaborative recommendation

The algorithm searches the most similar users from the user’s friends list. The profile of those users is used to provide more sophisticated suggestions to the target

user.

SRP REQ9

Provision of user clustering for creating videos addressed to specific groups.

User clusters are created based on business users’ queries in order to provide suggestions on specified

target group.

SRP REQ1

0

Export results in both a common format (e.g. excel) and IEVC compatible one

Results are produced in JSON format. More formats could be supported if needed.

SRP REQ 11

Provide suggestion on enrichments for either query or inferred target group

No mapping on tool’s functionality

SRP REQ

Provide level of preference and ranking about each

Each suggestion is accompanied with a score that shows how close the suggested content is to the user’s



page 58

12 suggestion preferences.

SRP REQ 13

User interface for querying and data visualization

A GUI is provided so that the user can interact with the tool.

SRP REQ 14

User query protocol for targeted data visualization

The GUI communicates with the backend of the tool. The responses are visualized in the GUI.

SRP REQ 15

Users' metadata involved: gender, age, education, country, occupation, type of enrichment

All the required users’ metadata are used in providing creating the clusters for the target group

recommendation.

SRP REQ 16

Decision interface for the user validation of the system output (suggestion)

The GUI give the user the ability to decide if he will accept the provided suggestion or not.

SRP REQ 17

Automatic interfacing with IEVCT tool

The interfacing with the IEVCT tool will be decided in the integration phase.


The SRP tool’s functionalities mainly focus on the post-production phase of the documentary creation procedure. The PRODUCER platform can make use of its functionalities in order to provide personalized suggestions to the audience members willing to consume its content. While the available content can be of vast volume, the recommendation engine can filter out the uninteresting part so that the viewer can efficiently navigate to the most relevant to his preferences content and thus vastly increasing his QoE. What’s more, by applying the most preferable enrichments, video watching can be transformed from a dull to an exciting experience.

An additional functionality the tool provides that the PRODUCER platform can make use of, is the recommendation of content to target groups. The tool gives the ability to make suggestions on content relevant to specified audience groups that a professional user is willing to focus on when creating his documentary. Even after the creation of the documentary, the tool can be used to apply different layers of enrichments, so that varied “versions” of the documentary can be provided to make it more appealing to different audiences. The above functionality is possible by making use of the information collected by the SRP tool from its audience members.

The main advantages of SRP tool are:

● It is a recommendation engine tailored to the PRODUCER’s content.



page 59

● It is an open source system so anyone can deploy it in his own platform and contribute to it’s codebase.

● It enables recommendation of enriched content. ● It enables the production of different layers of enriched documentaries ● It dynamically updates user profiles based on their interactions, allowing meaningful

inferences on specified target groups’ preferences as well as adapting as user preferences change over time.

6.4 Architectural Diagram & Placement in the overall architecture

In Figure 1 we can see the architecture of the tool.

Figure 1: Architectural diagram of SRP tool

Queries as well as user interactions are collected from the frontend of the platform and sent to the backend where further computations take place. Based on the data sent, the content management module and the user profile management module update the corresponding database entries, and according to the queries made by the users, the recommendation engine returns a filtering and ranking of the content to be presented. Further information about the tool as well as the algorithms used can be found in the following chapters.


Core Implementation Technologies

Python 3.6



page 60

Additional technologies utilised

Django 1.10

Django Rest Framework 3.5.1

Gensim library 0.13.4.1

Bootstrap – flat UI

uwsgi 2.0.15

Nginx 1.13

Docker

Database details PostgreSQL 10.1

Exposed APIs REST

Exchanged data format

JSON

GUI description GUI application communicating with the backend of the tool. Users have to signup/login to use the tool’s backend functionalities.


In Figure 1 the functional architecture of the SRPT tool is illustrated. The SRPT tool is realised by a frontend GUI and a backend service. The blue arrows are illustrating the Interactions and the dependencies among the components. Indicative workflow diagrams are provided in section .8. The tool is being developed as an open source project so the source code can be found in the following URL: https://github.com/vinPopulaire/SRPtool

6.7 Implementation

6.7.1 Graphical Interfaces

The Social Recommendation & Personalization tool provides a GUI in order to make it accessible to users willing to use the standalone version of the tool. In the integrated platform, the GUI is supposed to be part of the platform and better exploit the potential of the tool by combining its services with that of the other tools.

Upon hitting the URL of the tool, a greeting page is presented to the user (Figure 2).

https://github.com/vinPopulaire/SRPtool



page 61

Figure 2: login screen

There, the user can choose to see more information about the service by hitting the “About” or the “Terms & conditions” links, or login to the platform. If he/she already has credentials he/she can login from the page, or if not, choose to sign up by clicking on “Join Us”.

If he/she decides to sign up to the platform, a signup page will appear in order to fill some demographics information about himself/herself such as his/her name, surname, age, gender etc. (Figure 3). The information is used to initialize the user profile but will also be valuable when willing to gather information for a specific target group. When the user enters his/her information, the data is stored in the SRP tool database.



page 62

Figure 3: Sign up screen

After logging in, the user has the ability to change his/her personal information as he/she sees fit. By choosing “Profile” from the dropdown selection, he/she is presented with a form, prefilled with his/her information which can be modified (Figure 4). From the same dropdown list, he/she can choose to “Delete” his/her account or “Logout” if he/she wishes to.



page 63

Figure 4: Update Profile page

By clicking on “Videos” from the navigation bar, a list of videos is presented to the user (Figure 5). The list of the videos contains the top ten videos from the video database, ranked based on the profile of the user that requested the list by making use of the hybrid recommendation mechanism the tool utilizes. It is thus subject to change every time the user interacts with the system, so that the top videos correspond to what the system believes are the most interesting videos for the user at any time. In the Recommended Videos page, the user can click on the name of the video so that some text expands containing more information concerning the video, or choose the play button in order to watch the selected video.



page 64

Figure 5: Recommended Videos page

The “Play Video” page contains more information about the video, as well as the video itself (Figure 6). From this page, the user can view the video, interact with it by sharing it to social media, like it or dislike it and watch the enrichments associated with the video. In the right of the video, the recommended enrichments according to the profile of the user are presented and the user can click on them for additional information. The time in which they appear as well as the ability to share them is also present. All information concerning the interactions of the user with the content is sent back to the SRP tool backend to update the profile of the user in order to be able to make more precise recommendations in the future.



page 65

Figure 6: Play video page

The last page provided by the GUI is to be used by the content providers or producers willing to use the services provided by the SRP tool (Figure 7). The page is split in three columns. The leftmost contains a form where the user can select the audience group he/she wants to target in his/her documentary, so that the tool knows what recommendation to make. After choosing the appropriate values in the form, the user clicks on search and in the middle column, a list of the 10 most recommended videos for the target group appears. The list is ranked from most to least relevant. The user can once again click on the name of the video so that more information about the video is shown. After deciding on the appropriate video, by clicking the right arrow, the enrichments of the video appear on the right column. The enrichments are sorted based on the time they appear and only the most relevant enrichments for the target group at each time are shown. The tool gives the user the ability to select which ones of the suggested enrichments he/she finds appropriate for his/her documentary by toggling the slider at the top right of the enrichment. After making his/her selection, the user can export his/her choices for further use in the documentary creation process. In the integrated platform, the exported data could be used by the rest of the tools of the PRODUCER platform.



page 66

Figure 7: Business page

6.7.2 Algorithms

In this section, we will describe in depth the algorithms used by the tool, in order to provide meaningful recommendation to its audience users as well as content creators.

6.7.2.1 Initialization

As mentioned earlier, the SRP tool represents users as nodes in a graph, connected with term nodes, that correspond to the feature terms with which the multimedia content is represented. In our work, the feature terms used come from the upper layer of DMOZ [17], an attempt to create a hierarchical ontology scheme for organizing sites. Since the videos in the PRODUCER project are of generic nature, a common ontology scheme seems fit. The feature terms used are the following:

• Arts

• Business

• Computers

• Education

• Games

• Health

• Home



page 67

• News

• Recreation

• Science

• Shopping

• Society

• Sports

• Kids & teens

The weight on the edges of the user node to the terms node, signify the importance of each term to the user. By gathering information on the interactions of the user, the weight on those edges is updated to more precisely represent the interests of the user at the time based on which the SRP tool makes the content based recommendations.

The demographic information the user provides are used for collaborative recommendations purposes, and mainly for creating clusters of users in order to make recommendations for a particular target group. Thus, no node is created for that kind of information, and they are stored as a different part of the user profile in the SRP tool database.

The set of features used in our algorithms is denoted as 𝑃 = {𝑃1, 𝑃2, … , 𝑃𝑁} with cardinality 𝑁. Every user profile with have the same set of features but will vary on the weight they have on each one of them. The weights of the features are expressed as a vector

𝑊 = [𝑊𝑃1, 𝑊𝑃2

, … , 𝑊𝑃𝑁] , 𝑊𝑃𝑖

≥ 0, ∀𝑖

where 𝑊𝑃𝑖 is the weight of node 𝑃𝑖 for the specific user. As a result, each user will have a

different 𝑊 denoting his preferences and interests.

When the user first creates his account, the tool has no information about the preferences of the user, thus the vector that corresponds to that user profile is 𝑊 = [0, 0, … ,0] i.e. zero

weights on every edge. The demographics of the user are explicitly provided by the user upon sign up and are stored in the database for later use. If the user chooses to login using an online social network account, the information could be extracted from the user’s public profile, and the corresponding information would be prefilled on his profile.

6.7.2.2 Multimedia Content Indexing

The indexing of the multimedia content plays a really important part in the framework [18][19] since we need to have an accurate modelling of the content in order to be able to provide meaningful suggestions to users. If the modelling is erroneous, the recommendation engine will not be able to provide appropriate content to the user even if its knowledge of the user’s interest is precise. The feedback the user provides when interacting with the content shows what set of features the users prefers or does not like. The indexing of the multimedia content aims at expressing each item with a vector of weights of the feature nodes. That way, we have a similar representation of items and users which allows us to check for similarities between them. Following the previous notation, we denote a vector describing an item as

𝑋𝑃 = [𝑋𝑃1, 𝑋𝑃2

, … , 𝑋𝑃𝑁]

where 𝑋𝑃𝑖 is the weight of node 𝑃𝑖 for the specific item.



page 68

Each element of the vector 𝑋𝑃 needs to be generated in an automatic way from the metadata

that come together with the video since such a representation is not already available. To achieve this, a previous version of the tool used a tf-idf algorithm [20]. In the current version of the SRP tool, a more sophisticated approach is used. More specifically, the 𝑋𝑃 are appointed using the Paragraph Vector algorithm [21] a powerful machine learning technique for creating representations of text documents. The multidimensional vector representation created by the model is trained to be useful for predicting words in a paragraph. The paragraph vector is concatenated with several word vectors from the paragraph and predict the following word in the given context. Both paragraph vectors and word vectors are trained by the stochastic gradient descent and backpropagation [22]. While word vectors are shared among all documents, paragraph vectors are unique for each item. At prediction time, the paragraph vectors are inferred by fixing the word vectors and training the new paragraph vector until convergence. The text of the paragraph used by the SRP tool consists of the metadata that accompany the video, such as title, genre, summary, as well as annotations that are automatically extracted using object detection algorithms on the video.

Since the multidimensional vector representation is not useful to us in the way it is, we apply the same procedure on the feature terms used in our vector representations. That way, each feature term also has a multidimensional vector representation on the same space as the paragraphs. A similarity metric is applied between the paragraph and the terms, and we appoint the score as a value to the corresponding feature vector in our node representation.

In our algorithm we use a pre-trained model from the Wikipedia dataset which consists of millions of documents on a large variety of themes and as a result is a pretty generic dataset covering all the topics that are of interest.

6.7.2.3 Relevance Feedback

The Relevance Feedback mechanism is responsible to lead the user’s profile to convergence on his actual preferences, as well as to keep the user’s profile updated while the user’s interests change over time. As already mentioned, two types of relevance feedback are captured from the interactions of the user with the platform – implicit and explicit. Explicit feedback refers to the information the user gives concerning the content he consumes explicitly, such as pressing a like or dislike button or rating the content. Implicit feedback is the information that the tool collects based on the behavior of the viewer using the tool and the interactions he has with the content. Generally, explicit feedback is more accurate but not as preferable because of the time overhead it puts on the user, something that leads users to avoid providing such information.

The SRP tool makes use of both explicit and implicit user feedback in order to effectively generate user profiles. In order to select the implicit feedback that we would use in the SRP tool we carefully examined the possible interactions of the users with the multimedia content and chose the ones that may be meaningful in exposing his interests. On each signal sent by the user, we appoint a weight based on his interactions, that is used to update his generated profile. In the following table (Table 1) we state the implicit relevance feedback used by the SRP tool. It is interesting to note that the way the mechanism is implemented, it is relatively easy to add or remove implicit feedback signals if in the future, other signals are to be



page 69

considered as well.

Implicit Relevance Feedback Signal Weight Symbol

A selection of video 𝑊𝑎 ≤ 1

B playtime of video 𝑊𝑏 ≤ 1

C share of video 𝑊𝑐 ≤ 1

D click on enrichment 𝑊𝑑 ≤ 1

E share enrichment 𝑊𝑒 ≤ 1 Table 1: Implicit relevance feedback signals.

In Table 2 we can see how the weights of Table 1 are computed. An explanation of the weight used in each case is provided in the first column.

Explanation in weight computation Weight

A If the video is selected, the user seems to have some interest in it. 𝑊𝑎 = 1

B Let 𝑃 be the playtime of the video meaning the time he has been watching the video and T the total duration of the video. The weight is the percentage of the video he has actually watched.

𝑊𝑏 =𝑃

𝑇

C If the user choses to share the video, he is interested in it 𝑊𝑐 = 1

D Let #𝑒𝑛 be the number of enrichments the user clicked on the video,

and 𝑋 the total number of enrichments. The weight is the percentage of the enrichments the user clicked.

𝑊𝑑 = #𝑒𝑛

𝑋

E Let #𝑒𝑛 be the number of enrichments the user shared on the video, and 𝑋 the total number of enrichments. The weight is the percentage of

the enrichments the user shared.

𝑊𝑒 =#𝑒𝑛

𝑋

Table 2: Values of weight on implicit feedback signals

Apart from the implicit feedback, the SRP tool makes use of the explicit feedback as well. More specifically, the user can send three levels of interest on the video:

𝑊𝑙𝑒𝑣𝑒𝑙 = {+1 , 𝑙𝑖𝑘𝑒

−1, 𝑑𝑖𝑠𝑙𝑖𝑘𝑒0, 𝑖𝑛𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑡

Note that while the like and dislike signals are chosen by the user after watching the video, the indifferent signal is automatically generated when the user fails to provide one. This makes the explicit feedback action optional.

After taking all the above signals into account, a weight 𝐾𝑢 is calculated, expressing cumulatively all the relevance feedback collected. This cumulative weight is used to update the user profile on the SRP tool database. More specifically, we compute 𝐾𝑢 as follows:

𝐾𝑢 =𝑅𝑎

𝑥𝑊𝑎 + 𝑅𝑏𝑥𝑊𝑏 + 𝑅𝑐

𝑥𝑊𝑐 + 𝑅𝑑𝑥𝑊𝑑 + 𝑅𝑒

𝑥𝑊𝑒 + 𝑅𝑓𝑥𝑊𝑙𝑒𝑣𝑒𝑙

𝑅𝑎𝑥 + 𝑅𝑏

𝑥 + 𝑅𝑐𝑥 + 𝑅𝑑

𝑥 + 𝑅𝑒𝑥 + 𝑅𝑓

𝑥



page 70

where the vector 𝑅𝑥 = [𝑅𝑎𝑥 , 𝑅𝑏

𝑥 , … , 𝑅𝑓𝑥] represents the importance value assigned in each of the

feedback signals. Based on the evaluations in [23] a vector of 𝑅𝑥 = [10,10, … , 10].

Another interesting thing to note in the above equation is the way each weight is added in the computation of the 𝐾𝑢. The fact that the denominator is constant while in the nominator each weight is added separately, allows the online calculation of 𝐾𝑢 – there is no need to have the

whole bunch of the relevance feedback signals but the calculation can be done every time a signal arrives.

6.7.2.4 User Profiles Update

The 𝐾𝑢 computed earlier is used to update the weights of the edges between the user node

and the feature nodes, and as a result update the user profile generated by the tool, representing the user’s interest on the different topics. In order to the update, first a new weight is calculated based on the previous one and the interactions of the users on the multimedia content

𝑊𝑃𝑖

𝑁𝑒𝑤 = {𝑊𝑃𝑖

𝑂𝑙𝑑 + 𝐾𝑢𝑋𝑃𝑖}

0

∞

The above calculation takes into account the number of times the content is viewed since the calculation takes place additively. The result is projected within the space [0, ∞).

The 𝑋𝑃𝑖 used in the equation is a convex combination of the vectors of the multimedia content.

Since the content used in the SRP tool consists of videos and enrichments, the vector element 𝑋𝑃𝑖 is calculated as

𝑋𝑃𝑖= 𝑦1𝑋𝑃𝑖

𝑣 + 𝑦2𝑋𝑃𝑖

𝑐𝑒 + 𝑦3𝑋𝑃𝑖

𝑠𝑒

where 𝑋𝑃𝑖𝑣 is the value of the weight of the video’s term, 𝑋𝑃𝑖

𝑐𝑒 and 𝑋𝑃𝑖𝑠𝑒 are the value of the

weight of the clicked enrichment’s and shared enrichment’s term respectively and 𝑦1 + 𝑦2 +𝑦3 = 1. If more than one enrichment is clicked, the 𝑋𝑃𝑖

𝑒 is calculated as the average weight on

all the clicked enrichments. In the SRP tool, we have set the values of 𝑦1 = 0.8, 𝑦2 = 0.1 and

𝑦3 = 0.1.

After the calculation of 𝑊𝑃𝑖

𝑁𝑒𝑤, a normalized value is calculated by finding the maximum among

the 𝑊𝑃𝑖 and thus

𝑊𝑃𝑖

𝑁𝑒𝑤 =𝑊𝑃𝑖

𝑁𝑒𝑤

max𝑖

𝑊𝑃𝑖

6.7.2.5 Recommendation mechanisms

The content based approach makes use of the user profile generated using the previous mechanism, as well as the vector generated for the multimedia content by leveraging information about its content and characteristics.

When the tool tries to make suggestions for a specific user, his profile vector is compared to that of all the items in the repository and the relevance score for each item is computed. The similarity measure used to find the relevance can be any type of similarity metric that fits the



page 71

use case (e.g. inner product, cosine, Euclidean etc.). The SRP tool makes use of the cosine and the Euclidean similarity metric. More formally, in the case of the cosine similarity metr ic between the user and the product, the calculation takes the form

𝑠𝑖𝑚𝑢𝑝𝑐𝑜𝑛𝑡𝑒𝑛𝑡(𝑖, 𝑗) =

𝑈𝑖 ∗ 𝑋𝑗

‖𝑈𝑖‖‖𝑋𝑗‖

where 𝑈𝑖 is the weight vector of the user’s profile, and 𝑋𝑗 is the weight vector of the item.

The content based recommendation’s drawback is that it is overspecialized by nature, and so it tends to recommend similar items that match best the user’s profile. The tool tries to solve the problem by making use of the collaborative based recommendation mechanism as well. The collaborative filtering mechanism makes use of the group’s intelligence in order to provide meaningful recommendations to a user. The first step is to find users similar to the user in question that could most probably share common interests. Instead of searching over all users in the SRP tool database, the tool searches on their friends network of the user (should one exist e.g. from the ABT tool) so that the recommendations make more sense in a social perspective: users will most probably prefer watching something that their friends would prefer. To find the most similar user, we compute the similarity between the users, and in the case of cosine similarity

𝑠𝑖𝑚𝑢𝑢(𝑖, 𝑗) =𝑈𝑖 ∗ 𝑈𝑗

‖𝑈𝑖‖‖𝑈𝑗‖

where 𝑈𝑖 is the weight vector of the user’s profile and 𝑈𝑗 is the weight vector of the friend’s

profile.

After the calculation takes place, we select the 𝑛 more similar users to the target user and denote them as neighbors. The similarity between the user 𝑖 and the item 𝑗 in the collaborative

filtering approach is thus calculated as

𝑠𝑖𝑚𝑢𝑝𝑐𝑜𝑙𝑎𝑏𝑜𝑟𝑎𝑡𝑖𝑣𝑒(𝑖, 𝑗) = ∑ 𝑠𝑖𝑚𝑢𝑢(𝑖, 𝑠) ∗ 𝑠𝑖𝑚𝑢𝑝

𝑐𝑜𝑛𝑡𝑒𝑛𝑡(𝑠, 𝑗)

𝑛

𝑠=1

Finally, in order to calculate the final similarity score using the hybrid approach of both content and collaborative filtering mechanisms, we apply a convex combination of the above similarity values as following:

𝑠𝑖𝑚𝑢𝑝(𝑖, 𝑗) = (1 − 𝜃)𝑠𝑖𝑚𝑢𝑝𝑐𝑜𝑙𝑙𝑎𝑏𝑜𝑟𝑎𝑡𝑖𝑣𝑒(𝑖, 𝑗) + 𝜃𝑠𝑖𝑚𝑢𝑝

𝑐𝑜𝑛𝑡𝑒𝑛𝑡(𝑖, 𝑗), 0 ≤ 𝜃 ≤ 1

where the parameter 𝜃 is set to 0.5 as proposed in [23].

Based on the type of items the tool is supposed to recommend, a slightly different approach is used. As far as the videos are concerned, after the calculation of the score, the videos are ranked and the top 𝑘 items are presented to the user. On the other hand, for the enrichments,

two ways of recommendation are available. The tool can either rank and suggest the top 𝑘 enrichments just as before, but since the number of enrichments is arbitrary and having many enrichments in the same video does not really affect the user’s viewing experience, this doesn’t seem to be valuable in terms of QoE.for the user. That is why the tool can also calculate the score of all enrichments and present to the user only the best enrichment on



page 72

each annotation, provided that an annotation has multiple enrichments on top of it.

6.7.2.6 Target group recommendations

Apart from the recommendation of items to audience members, the SRP tool provides recommendations of multimedia content to specified target groups by professional users as well. In order to be able to achieve such a task, a different approach of recommendations should be considered.

In order to specify the target group, the tool gives the user a form where he can choose the characteristics of the audience he wants to focus on. Based on the choices he makes, the tool filters the database to find the matching users from the dataset it has collected. Since users having the aforementioned characteristics can have multiple different preferences and thus create multiple groups sharing common interests, a clustering is applied on the users based on their user profile. The algorithm of choice for the SRP tool is the k-means algorithm [16].

After the clustering, each cluster is represented by the clusterhead appointed by the algorithm, thus having a profile vector averaging the profile vectors of the users belonging in the cluster. The tool returns a list of clusterhead profiles together with the recommendation of multimedia content for each of them and the number of users belonging in the cluster as shown in Table 3. That way, the user can decide which bunch of recommendations to use based on the profile of the clusterhead and the number of members of the cluster.

# members Feature vector of cluster

Representative 1 𝑛1 𝑈𝐶1 = [𝑈1, 𝑼𝟐, … , 𝑈𝑁]

Representative 2 𝑛2 𝑈𝐶2 = [𝑈1, 𝑼𝟐, … , 𝑈𝑁]

… … …

Representative n 𝑛𝑛 𝑈𝐶𝑛 = [𝑈1, 𝑼𝟐, … , 𝑈𝑁]

Table 3: Result of recommendation to target group

Since the clusterheads are represented in the same way as the common users of the system, the recommendation mechanism used to suggest the appropriate content is the same as previously described.

6.8 How to access the tool

The frontend of the tool is accessible in the following URL:

https://sptool.netmode.ntua.gr/

where the user can login and interact with the tool via it’s GUI. Some simple use cases on how to use the tool and the services it provides can be found in the following chapter.

https://sptool.netmode.ntua.gr/



page 73


6.9.1 Audience member workflow

In the workflow of the Social Recommendation & Personalization tool in Figure 8, the user is a member of the audience willing to interact with the tool and make use of the services it provides.

Figure 8: Workflow of audience member using the SRP tool.

Step 1

The user enters the platform and the home screen appears. There he can choose to sign up or login if he already has an account. The user can choose to login using the SRP tool’s credentials or via a social platform (e.g. Facebook). Upon signup, and after providing some personal information (1a), the tool creates a user profile for the user that is going to be used in order to provide meaningful recommendations (1b).

Step 2

The user asks for a recommendation of videos he might like from the platform (2a). Upon request, the backend of the platform uses his user profile as well as other information about the user (such as his friends) and provides him with a recommendation of videos (2b).



page 74

Step 3

The user selects a video from the recommended list that seems interesting for further viewing (3a). The tool keeps track of the choice of the user and provides enrichments associated with that video that the user would find interesting (3b).

Step 4

The user watches the video and interacts with it by pressing play/pause buttons, sharing it or clicking on some enrichments that he finds interesting. The SRP tool collects all that information and updates the user’s profile so that it can make more accurate recommendations in the future (4b).

6.9.2 Content creator workflow

In this workflow, the user is not an audience member but instead a content creator or publisher, willing to create a documentary. The tool allows him to choose the content that is most relevant to his target group. The workflow can be seen in Figure 9.

Figure 9: Workflow of content creator using the SRP tool.



page 75

Step 1

After logging in the tool, the user states the demographics of the audience group he wants to target (1a). The SRP tool uses the data it has collected from its users in order to create clusters corresponding to the desired target group (1b). It then passes the profiles generated by the clustering through its recommendation engine and provides a list of recommended videos to the user (1c).

Step 2

The user selects the video willing to use on his documentary (2a). The backend of the tool, following the choice of the user, suggests the appropriate enrichments of the video for the target group specified (2b).

Step 3

Having the list of the suggested enrichments, together with the information of what the enrichment is about, as well as when it appears in the video, the user can select the ones that in his personal view are more appropriate for his content (3). He can thus accept or reject the suggestions of the tool by using the corresponding sliders.

Step 4

After making his selections, the user can export his selection of the enrichments for the selected video for further use in creating his documentary (4).


Wikipedia and Google News datasets where used as trained models for the Paragraph Vector algorithm.

Enriched videos from the MECANEX project [24] were used to test the tool’s main functionalities.

For the development of some features, such as friend networks, mockup data were used.



page 76

7 References

[1] K. Z. Gajos, D. S. Weld, J. O. Wobbrock, “Automatically Generating Personalized User Interfaces with

Supple”, Journal Artificial Intelligence Archive, Vol. 174, No. 12-13, pp. 910-950, August 2010.

[2] A. Kobsa, “Generic User Modeling Systems”, Journal of User Modeling and User-Adapted Interaction Vol.

11, No. 1-2, pp. 49-63, March 2001.

[3] U. Hanani, B. Shapira, P. Shoval, “Information Filtering: Overview of Issues, Research and Systems”, User

Model. User Adapt. Interact. Vol. 11, No. 3, pp. 203–259, 2001.

[4] M. Pazzani, D. Billsus, “Learning and Revising User Profiles: the Identification of Interesting Web Sites”,

Machine Learning, Vol. 27, No. 3, pp. 313–331, 1997.

[5] G. Salton, C. Buckley, “Improving Retrieval Performance by Relevance Feedback”, Jour of the American

Society for Information Science, Vol. 41, No. 4, pp. 288–297, June 1990.

[6] J. Ponte, “Language Models for Relevance Feedback”, in Advances in Information Retrieval, ed. W.B. Croft,

pp. 73-96, 2000

[7] R.W. White, J.M. Jose, C.J. van Rijsbergen, I.G. Ruthven, “A Simulated Study of Implicit Feedback Models”,

in Proc. of the 26th European Conference on Information Retrieval, Sunderland, UK, April 5-7, 2004.

[8] C. Buckley, G. Salton, J. Allan, “The Effect of Adding Relevance Information in a Relevance Feedback

Environment”, In Proc. of the 17th annual International ACM SIGIR Conference on Research and

Development in Information Retrieval (SIGIR '94), 1994.

[9] J. Bobadilla, F. Ortega, A. Hernando, A. Gutierrez, “Recommender Systems Survey”, Knowledge-Based

Systems, Elsevier, Vol. 46, pp. 109-132, April 2013.

[10] R. Meteren, M. Someren, “Using Content-Based Filtering for Recommendation”, in Proc. of ECML 2000

Workshop: Maching Learning in Information Age, pp. 47-56, 2000

[11] M. Pazzani, “A Framework for Collaborative, Content-Based, and Demographic Filtering”, Artificial

Intelligence Review-Special issue on Data Mining on the Internet, Vol. 13, pp. 393-408, 1999.

[12] B. Schafer, D. Frankowski, J. Herlocker, S. Sen, “Collaborative Filtering Recommender Systems”, The

Adaptive Web Chapter 9, Springer, pp. 291-324, 2007

[13] J. Bobadilla, F. Ortega, A. Hernando, A. Gutierrez, “Recommender Systems Survey”, Knowledge-Based

Systems, Elsevier, Vol. 46, pp. 109-132, April 2013.

[14] C. Porcel, A. Tejeda-Lorente, M. A. Martinez, E. Herrera-Viedma, “A hybrid Recommender System for the

Selective Dissemination of Research Resources in a technology Transfer Office”, Information Systems, Vol.

184, No. 1, pp. 13-32, 2012.

[15] S. Chen, S. Owusu, L. Zhou, “Social Network Based Recommendation Systems: A Short Survey",

International Conference on Social Computing (SocialCom), pp. 882-885, Sept. 2013.

[16] S. P. Lloyd (1957). Least squares quantization in PCM. Technical Report RR-5497, Bell Lab, September

1957.

[17] http://dmoztools.net/

[18] A. Micarelli, F. Sciarrone, “Anatomy and Empirical Evaluation of an Adaptive Web-based Information

Filtering System”, User Modeling and User-Adapted Interaction, Kluwer Academic Publishers, Vol. 14, pp.

159-200, 2004.

[19] G. Gentili , A. Micarelli , F. Sciarrone, “Infoweb: An Adaptive Information Filtering System for the Cultural

Heritage Domain”, Applied Artificial Intelligence, Vol. 17, No. 8-9, 2003.

[20] C. D. Manning., P. Raghavan,, H. Schütze. "Scoring, term weighting and the vector space model."

Introduction to information retrieval 100 (2008): 2-4.

[21] Q. Le and T. Mikolov, “Distributed representations of sentences and documents”, 31st International

Conference on Machine Learning, pp. 1188–1196, 2014.

[22] D.E Rumelhart, G.E. Hinton, and R.J. Williams. "Learning representations by back-propagating errors."

nature 323.6088 (1986): 533-536.

http://dmoztools.net/



page 77

[23] E. Stai, S. Kafetzoglou, E.E. Tsiropoulou and S. Papavassiliou, 2016. A holistic approach for personalization,

relevance feedback & recommendation in enriched multimedia content. Multimedia Tools and Applications,

pp.1-44.

[24] http://mecanex.eu Horizon 2020 research and innovation programme under grant agreement No 645206

[25] https://www.w3.org/TR/presentation-api/, Presentation API, W3C Candidate Recommendation 01 June 2017

[26] https://www.hbbtv.org/resource-library/#specifications, HbbTV 2.0.1 (TS 102 796 V1.4.1) Specification.

http://mecanex.eu/

producer · (pre-production, production & post-production tools) task no. t3.3 task title...

Documents