use cases & requirements -...

© ICoSOLE consortium: all rights reserved page i

Use Cases & Requirements

Deliverable D2.2

Work package / task: WP2 / T1

Document status: Draft

Confidentiality: Public

ICoSOLE identifier: ICoSOLE-D2.2-BBC-UseCasesAndRequirements_v15.docx

Deliverable number: D2.2

Main authors of Deliverable: Rik Bauwens (VRT), Mike Matton (VRT), Chris Pike (BBC), David Marston (BBC)

Internal reviewer: G. Kienast (JRS)

Version Date Reason of change

1 2014-05-30 Document created, initial structure

2 2014-06-02 Added and updated use case descriptions and requirements

3 2014-06-10 Further improvements to use case descriptions

4 2014-06-10 Added example service requirement table and UC5.2a details.

5 2014-06-12 iMinds updates to UC descriptions

6 2014-06-16 Changed UC names, incorporated TaW changes, added Venue Explorer UC, updated requirements table.

7 2014-06-17 Updated requirements types and status, new requirements added

8 2014-06-18 Updated UC diagrams for UC-4.1 and UC-4.2

9 2014-06-19 Adding UC-3.7 diagram. General editorial improvements throughout document.

10 2014-06-20 Adding priorities section

11 2014-06-27 Added service sheets, references between UCs and Rs, summary and intro, UI list and non-functional requirements, some activity charts

12 2014-07-01 Version for internal review

13 2014-07-07 Internal review comments

14 2014-07-28 Updated version after review

15 2014-07-30 Final version

Version of 2014-07-30 D2.2 Use Cases and Requirements

© ICoSOLE consortium: all rights reserved page ii

The work presented in this document was partially supported by the European Community under the 7th framework programme for R&D.

This document does not represent the opinion of the European Community, and the European Community is not responsible for any use that might be made of its content.

This document contains material, which is the copyright of certain ICoSOLE consortium parties, and may not be reproduced or copied without permission. All ICoSOLE consortium parties have agreed to full publication of this document. The commercial use of any information contained in this document may require a license from the proprietor of that information.

Neither the ICoSOLE consortium as a whole, nor a certain party of the ICoSOLE consortium warrant that the information contained in this document is capable of use, nor that use of the information is free from risk, and does not accept any liability for loss or damage suffered by any person using this information.


© ICoSOLE consortium: all rights reserved page iii

Table of Contents

List of Figures ........................................................................................................................................ v

List of Tables ........................................................................................................................................ vi

1 Executive Summary ......................................................................................................................... 1

2 Introduction ....................................................................................................................................... 32.1 Purpose of this Document .......................................................................................................... 32.2 Scope of this Document ............................................................................................................. 32.3 Status of this Document ............................................................................................................. 32.4 Related Documents .................................................................................................................... 3

3 List of Use Cases ............................................................................................................................. 43.1 UC-1 Production preparation ...................................................................................................... 43.2 UC-2 UGC Request & response platform................................................................................... 43.3 UC-3 Production application(s) ................................................................................................... 43.4 UC-4 Playback on TV ................................................................................................................. 43.5 UC-5 Playback on mobile device ................................................................................................ 4

4 Use Case Prioritisation .................................................................................................................... 54.1 Rationale for the BBC’s Priorities ............................................................................................... 64.2 Rationale for DTO’s Priorites ...................................................................................................... 64.3 Rationale for VRT’s Priorities ..................................................................................................... 74.4 Rationale for BIT’s Priorities ....................................................................................................... 74.5 Rationale for JRS’s Priorities ...................................................................................................... 74.6 Rationale for TaW’s Priorities ..................................................................................................... 84.7 Rationale for iMinds’ Priorities .................................................................................................... 8

5 Use Case Descriptions .................................................................................................................... 95.1 UC-1 Production preparation ...................................................................................................... 95.2 UC-2 UGC request & response platform .................................................................................. 185.3 UC-3 Production application(s) ................................................................................................. 265.4 UC-4 Playback on TV ............................................................................................................... 495.5 UC-5 Playback on mobile device .............................................................................................. 56

6 Requirements ................................................................................................................................. 696.1 Service Requirements .............................................................................................................. 696.2 User Interfaces ......................................................................................................................... 736.3 Non-Functional Requirements .................................................................................................. 73

7 Requirements Specifications ........................................................................................................ 747.1 Metadata Services .................................................................................................................... 747.2 Capture Services ...................................................................................................................... 817.3 Processing ................................................................................................................................ 867.4 Distribution services ................................................................................................................ 1007.5 Audience services ................................................................................................................... 106

8 Conclusions and Outlook ............................................................................................................ 116


© ICoSOLE consortium: all rights reserved page iv

9 References ..................................................................................................................................... 117

10Glossary ......................................................................................................................................... 118


© ICoSOLE consortium: all rights reserved page v

List of Figures

Figure 1: UC-1.1 diagram ......................................................................................................................... 10Figure 2: UC-1.2 diagram ......................................................................................................................... 11Figure 3: UC-1.3 diagram ......................................................................................................................... 13Figure 4: UC-1.4 diagram ......................................................................................................................... 14Figure 5: UC-1.5 diagram ......................................................................................................................... 16Figure 6: UC-1.6 diagram ......................................................................................................................... 18Figure 7: UC-2.1 diagram ......................................................................................................................... 19Figure 8: UC-2.2 diagram ......................................................................................................................... 21Figure 9: UC-2.3a high-level diagram ...................................................................................................... 23Figure 10: UC-2.3a diagram ..................................................................................................................... 23Figure 11: UC-2.3b diagram ..................................................................................................................... 25Figure 12: UC-2.3b Activity diagram ........................................................................................................ 26Figure 13: UC-3.1 diagram ....................................................................................................................... 28Figure 14: UC-3.2 diagram ....................................................................................................................... 30Figure 15: UC-3.2 diagram ....................................................................................................................... 30Figure 16: UC3.2 Activity diagram ............................................................................................................ 31Figure 17: UC-3.3 diagram ....................................................................................................................... 34Figure 18: UC3.3 Activity diagram ............................................................................................................ 35Figure 19: UC-3.4 diagram ....................................................................................................................... 38Figure 20: UC3.4 activity chart ................................................................................................................. 39Figure 21: UC-3.5 diagram ....................................................................................................................... 41Figure 22: UC-3.5 activity chart ................................................................................................................ 42Figure 23: UC-3.6 diagram ....................................................................................................................... 45Figure 24: UC3.6 activity chart ................................................................................................................. 46Figure 25: UC-3.7 diagram ....................................................................................................................... 48Figure 26: UC-4.1 diagram ....................................................................................................................... 52Figure 27: UC-4.2 diagram ....................................................................................................................... 55Figure 28: UC-5.1 high level diagram ....................................................................................................... 57Figure 29: UC-5.1 diagram ....................................................................................................................... 57Figure 30: UC5.1 Activity diagram ............................................................................................................ 58Figure 31: UC-5.2a diagram ..................................................................................................................... 60Figure 32: UC-5.2a diagram ..................................................................................................................... 60Figure 33: UC-5.2a Activity diagram ........................................................................................................ 61Figure 34: UC-5.2b diagram ..................................................................................................................... 64Figure 35: UC-5.2b activity chart .............................................................................................................. 65Figure 36: UC-5.2c diagram ..................................................................................................................... 67


© ICoSOLE consortium: all rights reserved page vi

List of Tables

Table 1: Use case priorities........................................................................................................................ 6Table 2: List of service requirements ....................................................................................................... 69


© ICoSOLE consortium: All rights reserved page 1

1 Executive Summary

Projects as large and as complex as ICoSOLE, have to be rationalised and broken down into realisable units. The team started with potential scenarios for the project, the primary one being a music festival, and the secondary being a sports event. From these scenarios a set of use cases were collected which cover every aspect of providing an immersive and interactive audio-visual experience of these large events to the consumer. The use cases were grouped into five categories: production preparation, UGC request & response platform, production applications, playback on TV and playback on mobile devices.

The production preparation use cases include managing the event calendar, location surveying, organising staff involved in the production and the equipment required for the shoot. As the project also involves contributions from the public attending the event (UGC - user generated content), it also required to be able to manage them, including providing them with the means to contribute their content.

The use cases for UGC request and response involve gathering the audio, video and images captured by the event attendees, which also includes sending out requests to them from the production team. The capturing of content from the event attendees will include generating useful information such as location and who or what is being captured; it will also require filtering out low quality or irrelevant material before being recorded. The 'Moments' mobile app will be used as a method for attendees for capturing action.

The production applications use cases cover everything involved in turning the audio and video captured both by the professionals and the public into something ready for broadcast or streaming. So this includes the professional capture from microphone arrays, TV and panoramic cameras, etc. These raw signals need editing so they are suitable for broadcasting, and also require metadata to be added and handled so the material can be suitably reproduced and recorded later on in the chain. As the experience will be both interactive (so consumers can adjust things like point-of-view and sound levels) and immersive, the metadata is a vital element to allow the audio and video to be rendered with this in mind.

The viewers at home will be watching the event on their TVs, so there are use cases to cover both live and on-demand playback for this. With TV reproduction the full immersive and interactive experience can be realised, including 3D audio if the speaker layouts are available.

Consumers wanting to watch the event on their mobile device are catered for in the last set of use cases. Both live and on-demand playback is handled, with extra features available through mobile apps including the 'Wall of Moments' which allows UGC content to be shared, and 'VenueExplorer' which provides a way of navigating around a multi-stage event. Immersive audio over headphones will be provided by using binaural rendering.

From the list of use cases a set of service requirements was drawn up. Service requirements are the next stage in planning, and divide the project into tasks that can be implemented by the team. Many services relate to multiple use cases. The requirements each fall into one of five categories: metadata, capture, processing, distribution and audience.

The different requirements within the metadata category cover the different types of metadata used in the project, including metadata for the event calendar, crew, technical equipment and so on. This metadata also has to be converted to a common standard and stored in an appropriate way.

The requirements for capture cover audio, video and UGC capture (so microphones, cameras and mobile devices), including calibration of equipment where needed (such as ensuring the cameras in multi-camera rigs are all matched).

The processing requirements cover both audio and video processing and editing, such as time-alignment, filtering and mixing of audio signals, and rendering audio signals for headphone and speaker playout. It also includes handling the scene representation (how all the audio and video are positioned in the event), and dealing with metadata required for that.

The distribution requirements deal with how the audio, video and data streams are handled, from capture to playout at the consumer end. This including synchronisation of signals, IP-based streams and live playout services.

The audience requirements provide services such as the playout platform, clients for live TV, control for the TV and the mobile apps used to explore events and view shared content.

So, this document gives full descriptions for 23 different use cases, and 42 difference service requirements and how they all interrelate. Therefore it will provide a reference point for the initial



technical work for the project. However, as the project evolves it is expected some of these use cases and requirements may change, or new ones will be needed. But they do give a solid starting point for the whole system to be developed from.



2 Introduction

2.1 Purpose of this Document This deliverable contains the detailed analysis of the usage scenarios into use cases; and functional and non-functional requirements.

2.2 Scope of this Document The list of use cases is derived from the described usage scenarios and extended according to new usage considerations. These use cases are described in detail.

A prioritisation of these use cases is given in order to inform the plan for implementation of required services.

A list of required services is extracted from the defined use cases and each requirement is described in detail.

Sequence diagrams show the application of required services when carrying out use cases.

A list of non-functional requirements has also been generated. Functional requirements define a specific system behaviour i.e. the system will do X. Non-functional requirements define a general system behaviour i.e. the system will be Y.

2.3 Status of this Document The ist the final version of D2.2.

2.4 Related Documents Before reading this document it is recommended to be familiar with the following documents:

D2.1 Usage scenarios



3 List of Use Cases

The list of use cases given in deliverable D2.1 Usage Scenarios has been updated and is presented here:

3.1 UC-1 Production preparation

UC-1.1 General administration

UC-1.2 Managing production

UC-1.3 Managing event calendar & locations

UC-1.4 Preparing archive material

UC-1.5 Access to UGC request & response platform

UC-1.6 Site survey for professional capture

3.2 UC-2 UGC Request & response platform


UC-2.2 Checking & choosing requests (preparation)

UC-2.3a Capturing (UGC)

UC-2.3b Moments

3.3 UC-3 Production application(s)

UC-3.1 Review production preparation during event setup

UC-3.2 Live editing

UC-3.3 Data & metadata management

UC-3.4 Advanced audio authoring and rendering

UC-3.5 Offline editing

UC-3.6 Professional capturing

UC-3.7 Advanced video authoring and rendering

3.4 UC-4 Playback on TV

UC-4.1 Live playback on TV

UC-4.2 On-demand playback on TV

3.5 UC-5 Playback on mobile device

UC-5.1 Live playback on mobile device

UC-5.2a On-demand playback on mobile device

UC-5.2b The Wall of Moments

UC-5.3c Venue Explorer



4 Use Case Prioritisation

A summary of how some of the partners prioritise each of the use cases is shown the table below. Three levels of priority are used (low, medium, high). The overall priority is taken as the highest priority given by any partner. An overall score is also noted to represent the group prioritisation; low, medium, and high priorities were given 0, 1, and 2 points respectively.

UC Title BBC DTO JRS VRT BIT iMinds TaW Overall

1.1 General administration

Low Low Low Low Low Low Low Low (0)

1.2 Managing production

Low Low Low High Low Low Low High (2)

1.3 Managing event calendar & locations

Low Low Low High Low Low Low High (2)

1.4 Preparing archive material


1.5 Access to UGC R&R platform

Low Low Low Med Low Low Low Med (1)

1.6 Site survey for capture

Low Low High Med Low High Low High (4)

2.1 General administration


2.2 Checking & choosing requests

Med Low Low Med Low Low Low Med (2)

2.3a UGC Capturing Med High Med High Med Low Low High (7)

2.3b Moments Low Med Low Low Med Low Low Med (2)

3.1 Review production preparation


3.2 Live editing High High Med High Low Med High High (10)

3.3 Data & metadata management

High Med High High Med Med Low High (9)

3.4 Advanced audio authoring & rendering

High High Low High High Low Low High (8)

3.5 Offline Editing Low Low Med High Low Med Low High (4)

3.6 Professional Capturing

High Med Med High Med High Low High (9)

3.7 Advanced video authoring & rendering

Medium

Low High High Med High Low High (8)

4.1 Live playback on TV

Med Low Low Low High High Low High (5)

4.2 On-demand Med Low Low Low High Low Low High (3)



UC Title BBC DTO JRS VRT BIT iMinds TaW Overall

playback on TV

5.1 Live playback on mobile

High High Low Med High Low Low High (7)

5.2a On-demand playback on mobile

High High Low High High Low Low High (8)

5.2b The Wall of Moments

Med Low Low High Med Med Low High (5)

5.2c Venue Explorer High Med Low Med Med Med Low High (6)

Table 1: Use case priorities

4.1 Rationale for the BBC’s Priorities

There are production planning and administration workflows already in place at the BBC. It is considered unlikely that ICoSOLE developments in this area would be transferred to a product that was used by BBC production teams. Therefore these use case areas (UCs 1.1-1.3) are ranked as low priority. Also given low priority is the UGC request and response platform (UC 1.5) and the site survey (UC 1.6). Whilst both of these components may be useful in the future, the core elements of the production and delivery of user experience need prioritisation.

The higher priority work covers the editing and mixing of professional and UGC content into an immersive (and potentially interactive) user experience. Data and meta-data management (UC 3.3) is an important use case, it highlights the need for underlying systems and data models required to represent large events with multiple sub-events, covering the management of the overall event model and its data streams.

Mobile rendering is given higher priority than TV rendering. The target platform for TV UCs needs defining to aid in prioritisation. However it could be argued that the mobile platform target constitutes a superset of the technical innovations required for the TV platform, since for mobile platforms the streaming solution should incorporate multiple events but also adaptive capabilities.

There is potential on both targets for interesting work on design of an efficient and user-friendly interface for navigating the event streams. However time and effort limitations mean that it is necessary to focus more heavily on development for one of these platforms.

Improved mobile experience is an important strategic area for the BBC. Immersive navigation of multiple event streams and interactive control of certain rendering features is deemed more relevant to the mobile case. There is also more scope for technical innovation on the mobile rendering side than for the loudspeaker rendering for audio.

4.2 Rationale for DTO’s Priorites

Being a Research & Innovation centre of Technicolor, DTO is mainly interested in investigating possibilities for the capturing of audio UGC and its fusion into live production.

The higher priority use cases cover the UGC capturing as well as the play-out on mobile devices. The priorities reflect the innovative approach of the ICoSOLE project aiming at bringing an immersive impression of live events even to mobile users.

In particular the capturing of audio user generated content and its automatic analysis with respect to quality evaluation, suitability for the production, and metadata generation ask for new processing techniques, currently not in use with conventional production work-flows. Moreover, the use of HOA based 3D audio format covering the whole chain of content capturing – including UGC as well as professional content –, processing (e.g. microphone fusion), mixing, lossy audio coding, up to presentation needs to be studied in more detail.



4.3 Rationale for VRT’s Priorities

4.3.1 Production preparation

Motivation: as the event timeline and event programme are crucial metadata in a central ICoSOLE metadata model, good tools to import / edit these are crucial. Further, solutions for preparing archive material already exist. Therefore, VRT does not see lots of innovation opportunities to add this as high priority. It could be a nice extension however.

Next, the ability to include UGC in professional content is considered important by the internal stakeholders at VRT. However, the first focus should be on setting the technical contribution and automated tagging right. The ability to “plan” UGC is of interest as well. Good to note is that the concept of trust and the ability to personally communicate between the professional team and the contributing user is considered very important.

4.3.2 UGC platform

The first focus should be on the capturing itself. The first goal is to allow smooth uploading of the content (including automated tagging, etc.). The ability to plan and schedule UGC capturing has slightly lower priority. Also the ability to send structured feedback to contributing users is considered important. Sharing this feedback on social media could be an important part of the “event experience” for the contributing users. They could gain a higher social status in a community by gaining an “approved by VRT” badge on their social media profile (e.g. Facebook, Twitter, Instagram), indicating that their content was used and considered of good quality by the network. This could provide additional publicity for the platform as well as stronger user commitment, leading to more valuable UGC of a higher quality.

4.3.3 Production

As stated before, catch-up is considered of slightly higher priority with respect to live. However, some parts of the live editing use case are very important for catch-up as well (automatic tagging, checking synchronization, automatically detect incidents etc. These have the highest priority).

Professional content capturing and all metadata-related topics are considered as the central part of the ICoSOLE system and considered very important for a broadcaster such as VRT.

4.3.4 Play-back on TV

As has been stated before, the main focus should be on mobile. Therefore, both UC-4.1 and UC-4.2 are low priority (could have if time is available).

4.3.5 Mobile play-out

Catch-up abilities for live events (especially music festivals) are considered slightly more important than the ability to stream live.

4.4 Rationale for BIT’s Priorities

BIT’s main areas of prioritisation for the ICoSOLE project are:

Adaptive playout application (desktop and mobile). As from our point of view the live and on-demand scenarios are quite similar, we will of course focus on both.

3D audio integration into MPEG-DASH

Cloud based transcoding

4.5 Rationale for JRS’s Priorities

JRS’ involvement in the project focuses on automating the selection of content for the production. Therefore, the use cases contributing to this aim have been given high priority. The first of these use cases is site survey, where measurements and recordings of the site of the event are collected, in order to use them as a reference for semi-automatic alignment of the media capture during the event. Data and metadata management happens during recording, and is concerned with organising all incoming media and metadata, including possible manual intervention for quality assurance. Based on automatically and manually extracted metadata, content filtering is performed. Finally, video authoring and rendering will make use of metadata to create the video scene, allowing the production team to



make final adjustments. The related use cases for capturing user generated content and for performing live editing have been given medium priority.

4.6 Rationale for TaW’s Priorities

TaW will be developing tools and interfacing for the live editing parts of the project. Therefore their priority is use case 3.2.

4.7 Rationale for iMinds’ Priorities

For the 360 degree and other professional video capture the use cases for professional video capture and site surveying will be important. This video will require metadata editing and management, hence the higher priority for that use case, along with live editing as well. For the video playback, rendering of the immersive video is also a high priority.



5 Use Case Descriptions

This section provides detailed descriptions of the use cases with use case diagrams. For each use case, functional requirements have been extracted and are listed as references to the services described in section 76.2. For certain use cases with more complex requirements, an activity chart is also provided.

5.1 UC-1 Production preparation

5.1.1 UC-1.1 General administration

Description

This use case describes the management of the production preparation platform. Members of the production crew login to the platform to manage productions. A new member can be added or removed too.

Actors

Producer

Production team member

Flow of events

1. Producer creates account on production preparation platform

2. Producer logs into the production preparation platform

3. Producer adds accounts for production crew members

4. Producer manages production

5. Production crew member logs into the production application

6. Production crew member manages production

Inputs

[User] Manual metadata input

Outputs

[System] All information is stored in a metadata store

Interfaces

Only metadata is exchanged

Priority

Low (0)



Use case diagram

Figure 1: UC-1.1 diagram

Requirements

R1.1 Reading & storing administrative metadata

5.1.2 UC-1.2 Managing production

Description

This use case describes the general management of preproduction. It contains the management of all production metadata and allows assignments of crew. It is directly coupled with the event calendar and event locations.

Innovation

Planning of complex production with many different inputs (professional camera, microphone, omnidirectional camera, UGC)

Dependencies


UC-1.3 Event calendar & locations (for being able to plan crew and technical equipment to event locations)

Actors

Producer


Flow of events

1. First the producer starts a new production.



2. Together with his team, he enters different metadata (both administrative and technical) about the production

3. Next, he plans the available production crew in the event. This includes selecting the technical equipment they will set up.

4. All information is stored into the ICoSOLE system (scene model)

Inputs


[User + System] Information from event calendar

Outputs

[System] All information should be stored in a metadata store. The equipment and crew information will become part of the ICoSOLE scene model.

Interfaces

Only metadata needs to be exchanged. Metadata format is to be decided.

Priority

High (VRT) (2)

Use case diagram


Requirements


R1.2 Reading & storing event calendar metadata

R1.3 Reading & storing technical equipment metadata

R1.4 Reading & storing production crew information

5.1.3 UC-1.3 Managing event calendar & locations

Description

In this use case, a producer or production team member can connect to an event calendar of a city festival or sports event and display it on a timeline and sorted by event location. He can browse through



the calendar and make a selection of events and locations for capturing. He can also add itineraries and participants for closed circuit sports events or festival line-ups.

Innovation

Preparing an event capture based on specific event timeline and locations.

Dependencies

UC 1.1 General administration

UC 1.2 Managing production

Actors

Producer


Flow of events

1. Producer logs into production preparation location

2. Producer creates a new production or selects a production

3. Producer connects to an event calendar related to the production

4. Producer or team member adds event information – festival line-ups, itineraries and sports participants

5. Producer selects events and locations of the festival for capturing

Inputs

[User] Production of an event

Outputs

[System] Event line-up, sports event itineraries and participants

[System] Selection of festival events and locations for capturing

Interfaces

Only metadata needs to be exchanged. Metadata format is to be decided.

Priority

High (VRT) (2)



Use case diagram


Requirements



5.1.4 UC-1.4 Preparing archive material

Description

This use case describes the preparation of archive material to use later on in a live or on-demand production. A production crew member searches for appropriate archive material (using semi- automatic filters) and adds it to the production.

Innovation

Semi-automatically filter appropriate archive material.

Dependencies


UC-1.3 Managing event calendar & locations

Actors

Production crew member



Flow of events

1. Production crew member logs in to the production preparation application

2. Based on the productions settings, genre, location etc., archive material is already automatically filtered ordered by relevance

3. Production crew member selects material to use later on in a live or on-demand production

Inputs

[System] Production metadata, archive material

Outputs

[System] Archive material for use in live or on-demand production

Interfaces

Metadata, data

Priority

Low (0)

Use case diagram


Requirements




5.1.5 UC-1.5 Access to UGC request & response platform

Description

In this use case, a producer can access the user generated content request & response platform from within the production preparation application. He can add requests for specific events / locations onto the platform and issue them.

Innovation

Collecting useful user generated content where professional capture is not possible.

Dependencies


UC 1.2 Managing production

UC 1.3 Event calendar & locations

Actors

Producer


Flow of events

1. Producer checks event calendar

2. Producer checks which events are not covered by a professional production crew

3. Producer opens content request & response platform and issues requests for these events

Inputs

[System] Event calendar

[System] Crew calendar

Outputs

[System] Requests on the platform

Interfaces

Metadata

Priority

Medium (VRT) (1)



Use case diagram


Requirements




R1.5 Reading & storing UGC capture requests

5.1.6 UC-1.6 Site survey for professional capture

Description

Carry out measurements to allow a reconstruction of a 3D model of the event location. We envisage taking geo-located photographs, panoramic photographs and using Google Streetview (see Glossary) imagery, etc. We also envisage using structures from motion techniques to reconstruct an image based model of the event location, similar to the Microsoft Photo-Tourism (see Glossary) system. This image based model of the event scenery "backdrop" will serve to track professional and UGC cameras during the event.

Innovation

Automation of aligning material taken at site survey

Use for improvement of localization accuracy of sensors

Dependencies

UC-1.2 Managing production (static metadata about scene setup and production context)

UC-3.2 Live editing (matching captured data against imagery from site survey)



UC-3.7 Advanced video authoring and rendering (use of model and aligned background images)

Actors

Location scout

3D modeller

Flow of events

1. Take measurements of the venue

2. Take still images, audio recordings and videos (e.g. pans through the venue) at defined locations

3. Collect third party material about the location (e.g. Streetview images)

4. Perform calibration

5. Build 3D model of location

6. Prepare background images for use in scene creation (advanced video authoring)

Inputs

[System] A set of video and image data

[User] A set of sound recordings

[User] Measurements, maps, etc.

Outputs

[System] Ground-truth localized images/video

[System] 3D model

[System] Aligned background images

Interfaces

Spatial metadata

3D model

Priority

High (JRS, iMinds) (4)



Use case diagram


Requirements




R2.1 Calibration of used technical instruments

R3.1 Automatic aligning of picture taken on site survey

5.2 UC-2 UGC request & response platform

5.2.1 UC-2.1 General administration

Description

This use case describes the management of the UGC platform. Users can create an account on the UGC platform. Thereafter, they can login to the UGC platform with the chosen credentials.

Actors

UGC contributors

Flow of events

1. UGC contributor creates account on UGC platform

2. UGC logs in to the UGC platform

Inputs




Outputs

[System] All information is stored in a metadata store

Interfaces

Only metadata is exchanged

Priority

Low (0)

Use case diagram


Requirements


5.2.2 UC-2.2 Checking & choosing requests (preparation)

Description

In this use case, a UGC contributor can check requests issued by a producer on the platform. Next, he can confirm to contribute to a particular request. An event will be added to his personal calendar and the producer will be informed of the intention.

Innovation

Set up of a platform where users can check on and contribute to requests from different events



Linking with personal calendar

Messaging system for users and contributors

Social media integration

Dependencies


Actors

UGC contributor

Flow of events

1. UGC contributor logs on to the platform and checks for requests for different events

2. UGC contributor confirms to contribute to a specific request

3. Event is added to user’s personal calendar

4. Producer is being informed of the contributor’s intention

Optional

5. UGC contributor connects his account on the platform to Facebook and Twitter

6. Once connected, the contributor shares his intention to contribute on Facebook and Twitter, along with a link to the platform and event

7. Friends of this contributor notice the post and are persuaded to contribute, too

Inputs

[System] Event calendar

[User] Requests by producer

[User] Interest from user

Outputs

[User] Contribution from UGC contributor

Interfaces

Metadata

Priority

Medium (2)



Use case diagram


Requirements



R1.8 Reading & storing UGC capture intentions

R1.9 Messaging system for users & producers

R1.10 Social media integration

5.2.3 UC-2.3a Capturing (UGC)

Description

This use case deals with capturing content from the UGC contributor (amateur film maker). It results in multimedia content (and metadata) which is handed over (and handled) by producers.

By starting the ICoSOLE app on the smartphone (tablet, etc.) and logging on, the system automatically captures time and location (section). Furthermore, an ICoSOLE accredited interview badge (probably a credit-card style ID card) is provided to the contributor. Content like interviews, warming-ups or sound-checks can be captured previous to the event and added to the appropriate section within the app. All content is supplemented with metadata, including geo-locations, recording parameters, or directional information as well as manual annotations of all kinds.

Captured content will be uploaded (on-the-fly) to a central storage and processing unit on site (typically in an outside broadcast van). Where filtered (in terms of quality) UGC content and professional content is stored. The UGC providers can receive messages from the producer (live editor) according to points of interests which are not well covered yet or hints for quality improvement of the recordings.



Innovation

Novel tools for capturing a consistent spatiotemporal representation of a spatially spread-out event from a wide variety of audio and video source

On-device analyses of captured essence and fusion with metadata from sensors

Algorithms for fusing audio and video content capture with heterogeneous devices into a format agnostic representation

Streaming approach with energy consumption in mind to extend the battery lifetime of mobile devices

Dependencies

UC-2.1 General Administration

UC-2.2 Checking & choosing requests

Actors

UGC contributor

Flow of events

1. The UGC contributor checks the platform for where to contribute

a. The UGC contributor confirms his/her contribution

b. The event is automatically added to his/her agenda

c. The Producer is informed of his/her intention

2. At the location the UGC contributor logs on to the system (e.g. with his/her smartphone), where time and location will be captured

3. The UGC contributor captures content, as well as metadata, with his/her device

a. Clips like interviews, etc. can be added to the dedicated section of the app

b. Also “non-technical” metadata can be added (e.g. comments)

c. The UGC contributor may receive inputs from the producer according to quality or location

4. The captured content will be uploaded to a central storage automatically

5. The UGC contributor ends his/her session by stopping recording

Inputs

[System] General information about the event on the platform

[User] Information on quality, location, etc. during capturing from producers

Outputs

[System] Content (captured audio/video)

[System] Metadata (automatic generated and user input)

Interfaces

To be decided (audio/video/metadata + feedback)

Priority

High (DTO, VRT) (7)



Steps (high level)

Figure 9: UC-2.3a high-level diagram

Use case diagram

Figure 10: UC-2.3a diagram

Requirements


R1.8 Reading & storing UGC metadata (comments, location, sensors, etc.)

R1.9 Messaging system for users & producers

R2.2 Capturing video

R2.4 Upload service

R3.14 Media coding

R4.2 Signal routing in production system

R4.3 Production data storage



5.2.4 UC-2.3b Moments

Description

This use case describes a mobile app called 'Moments', which a festival visitor can use to capture an interesting Moment with his/her smartphone (e.g. crowd-surfing by an artist). A Moment is a short A/V stream (approx. 30'') which is automatically tagged with location, orientation and time information (sensor metadata). A user can do some basic video editing and tagging as well (manual metadata). Finally, a Moment can be shared on social media and added to the ICoSOLE system. A remote user (e.g. at home or on the way) can access Moments via 'The Wall of Moments' app (UC-5.2b).

Innovation

Immersive and personalized A/V streaming such as FaceTime and AirPlay (see Glossary)

Merging and synchronizing user A/V stream with production A/V stream

Integrate social media to create more immersive experience (e.g. Wall)

Dependencies

UC-2.3a Capturing (UGC)

Actors

Festival participants

Flow of events

1. User opens the Moments app and taps ‘Record’ to start recording

2. A Moment is adaptively uploaded while capture is on-going

3. User taps ‘Stop’ when the Moment has passed

a. The Moment is automatically annotated with sensor information (e.g. location, orientation, timestamp)

b. User has basic editing tools for adding effects and labels to the video (e.g. to point to where the action is taking place by tapping the region in the video with his/her finger)

c. User can add additional tags (manual metadata, e.g. what happens in the Moment, who is present)

d. User can change which professional camera view should be synchronised with his/her Moment (e.g. director’s cut, camera on crowd, close-up camera)

4. User shares his Moment on the Wall (via social media)

Applied to live situation (these steps replace step 1 and 2)

1. Participant of the festival initiates A/V call (FaceTime) with someone at home

2. User at home accepts incoming call, can see what festival participant is capturing (video), hear the production audio and see the production video picture-in-picture

3. Step 3 from the previous scenario is initiated

Inputs

[User] A/V feed from participant device

[System] A/V feed from festival production


Outputs

[System] The Moment (A/V feed from mobile device, along with a link to the festival stream starting at the time the Moment was captured)



Interfaces

Metadata, data (A/V)

Priority

Medium (DTO, VRT) (2)

Use case diagram

Figure 11: UC-2.3b diagram

Requirements

R1.8 Reading & storing UGC metadata



R2.4 Upload service

R3.4 Timeline alignment of content

R3.7 Automatic extraction of metadata

R3.14 Media coding





Activity Diagram

Figure 12: UC-2.3b Activity diagram

5.3 UC-3 Production application(s)

5.3.1 UC-3.1 Review production preparation during event setup

Description

This use case describes the application that a producer uses to review the live production. He has a calendar/timetable overview of the different performances where he can see what is happening at a specific time or place. Also, a virtual map of the scene is shown, displaying the position of the different crews, as well as the type and position of all devices (cameras, microphones and so on). People who intend to contribute to the UGC programme are located on the map as well.

Innovation

Combining metadata from different sources and using it to make a time-aligned, location-aware representation.

Dependencies


UC-1.3 Event calendar & locations

UC-1.5 Access to user generated request & response platform

Actors

Producer

Production crew member



Flow of events

1. Producer signs in to production review application

2. Producer selects the timeline of the event, which shows the planning of the concerts, agenda of the crew and UGC participants and device information

3. Producer selects the virtual map of the event, an gets an overview of where the crew, devices and UGC participants are located

Inputs

[System] Event calendar & location

[User] Technical devices information

[User] Crew agenda information

Outputs

[User] Time-aligned & location-aware overview

Interfaces

Metadata

Priority

Low (0)



Use case diagram


Requirements





R1.5 Reading & storing UGC capture requests and intentions

R1.6 Reading & storing 3D model of event location

5.3.2 UC-3.2 Live editing

Description

Live editing allows the director and his staff to select, compose and schedule the content of the live programme. The editing application produces a linear broadcast which can be accessed by on-demand and enhanced media in later stages of the production. Input media are audio / video streams and other assets like graphics that are made available by previous recording and scene rendering.

Metadata information that is either carried with the media data themselves, or made available by metadata management, is used for filtering a selection of streams automatically.

Metadata streams carried with media data, for instance created by advanced media authoring processes (UC-3.4 and UC-3.7), are passed to the consumer endpoint along with the audio and video.



The audio and video streams can include both current formats (stereo/5.1 and HD) and new immersive formats (immersive object-based audio and panoramic/omnidirectional video).

Live editing also allows adding of simple graphics instantaneously to the production. The content of these graphics can depend on metadata, or other data streams to which the live editor has access (manually or automatically). Graphics templates for informative messages might have been prepared earlier during preparation of the production.

Live editing provides the editors with a global timeline and a time-coherent view of all incoming streams and media assets. Live production uses ad-hoc editing of the schedule and any eventual graphics content. This requires the underlying system to be able to perform in real time. However, the editing system also allows scheduling media on the global timeline later than the current live time. Off-site teams might be preparing a pre-scheduled segment, e.g. with a recap of the last two hours, using the same editing solution. The live editing system is therefore inherently a multi-user system.

Innovation

Strong integration of metadata into live content, production tools and delivery systems.

Hybrid scheduling mixing instantaneous and pre-scheduled production paradigms.

Unified handling of live assets (live streams) vs. offline assets (rendered files on storage).

Presenting relevant metadata with the content in a way suitable for live requirements (informative, reduced to the necessary)

Automatic filtering and pre-selection, adaptive to implicit feedback from production team.

Easy composition and overlay of simple graphics on top of the live stream.

Dependencies

UC-1.2: Managing production

UC-2.3a: UGC Capturing

UC-2.3b: Moments

UC-3.3: Data and metadata management UC-3.4: Advanced audio authoring and rendering

UC-3.7: Advanced video authoring and rendering

Actors

Editors

Director

Flow of events

1. An editor opens an instance of the editor

2. The editor inspects the available streams and assets with its metadata in the live editing UI.

3. With this information and the instructions of the producer, the editor selects and composes one or more output streams.

4. The producer continually observes the programme output, and possibly a real-time preview of the upcoming segments. The producer will give feedback to the editor (i.e. looping back to event 3).

Inputs

[System] Media streams with metadata and graphics from the asset storage system (including live feeds).

[User] User input by the editor.

Outputs

[System] One or more linearly-composed channels.

[System] Distribution of those channels is handled by the playout engine.



Interfaces

Graphical UI system (to be determined)

Backend communication to the playout engine (possibly REST, but needs to be determined)

A low-latency streaming format should be used for previewing available input streams as well as monitoring the live program output. RTMP would fit the requirements, although we would like to investigate using DASH for these purposes as well.

Priority

High (10)

Use case diagram





Activity Diagram

Figure 16: UC3.2 Activity diagram

Requirements


R1.8 Reading & storing UGC metadata (comments, location, sensors, etc.)

R3.2 Automatic filtering of streams

R3.3 Broadcast graphics service


R3.5 Live editing service

R3.8 Semi-automatic linking of content with metadata

R3.9 Audio processing

R3.10 Audio analysis

R3.11 Audio mixing

R3.12 Headphone sound rendering

R3.13 Loudspeaker sound rendering

R3.14 Media coding

R4.1 Synchronisation signal distribution



R4.4 Live playout service

R4.5 Scalable IP-based distribution scheme for live playback

R4.6 Adaptive bit-rate media delivery

5.3.3 UC-3.3 Data & metadata management

Description

This use case deals with ingesting, storing, organising and linking the data and metadata contributed before and in particular during the production. The result is that arriving content and metadata are integrated in the scene representation.

The use case consists of monitoring the back-end processes that put content and metadata onto the storage. The aim is to prepare content for selection and scene creation by putting into context, updating/correcting metadata and linking with other material.



Metadata is automatically extracted from the incoming content, and the content and metadata are put into context by semi-automatic linking with production data (e.g. the overall scene setup), and inserted into the scene representation make it accessible for selection and scene creation.

The actors, a metadata editor and a scene creator monitor an overview of the material arriving (cf. feed in traditional live production). The users may work on a specific view of the scene representation (e.g., filtered by location) rather than on the entire scene in some cases. The metadata captured with the content, added by UGC contributors or extracted automatically will be reviewed and possibly amended/corrected. Due to the constraints of live production the latter will often only consist in flagging metadata as reliable/seemingly correct or not. In addition, locations or key persons may be annotated.

A specific focus in metadata editing will be checking, and possibly adjusting, the spatial composition of a/v streams in the scene. For nearby or overlapping streams options for transitions can be set or modified.

Innovation

Enable handling metadata throughout the entire workflow

Automatic metadata extraction from a/v content

Semi-automatic linking between contributed content, production metadata and external data

Efficient user interfaces for review of content and metadata contributions, and interaction options for verification/correction/amendment of metadata

Dependencies

UC-1.2 Managing production (static metadata about scene setup and production context, external metadata sources)

UC-2.3a UGC Capturing (during event)

The use case provides input for:

UC-3.2 Live editing (content & metadata for selection)

UC-3.4 Advanced audio authoring and rendering (linked content & metadata)


UC-3.6 Professional capturing


Actors

Scene creator

Metadata editor

Flow of events

For each capture device added (professional or UGC)

1. Content arrives

2. Convert metadata to a common standard format (Audio Definition Model [EBU, 2013] for audio metadata) if required

3. Merge in any static metadata (e.g. schedules)

4. Check content (decide usable/unusable)

5. Review metadata (captured and automatically generated), in particular location

6. Add/update metadata (e.g. locations, salient actors/objects)

Inputs

[System] A set of audio and video streams

[System] A set of metadata streams

[System] Access to static metadata (spatial data about sets, schedules, playlists, etc.)



[User] Metadata annotations/updates made by metadata editor/scene creator

Outputs

[System] A set of (updated) metadata streams

[System] Metadata for content filtering/selection (not sure yet if those are streams as well)

[System] Storage of content and metadata (maybe just a “default” sink for the streams)

Interfaces

A/V streams for preview and time-based metadata visualisation

Metadata model(s) used in the streams from capture and automatic analysis

User interaction for annotations

Priority

High (9)



Use case diagram




Activity diagram


Requirements





R1.7 Convert metadata to a common standard


R3.1 Automatic aligning of picture taken on site survey



R3.6 Editing scene representation

R3.7 Automatic extraction of metadata





5.3.4 UC-3.4 Advanced audio authoring and rendering

Description

This use case concerns the creation of metadata required to represent and render an immersive audio scene within the production application, including monitoring of the rendered scene.

It involves selecting from the available audio source streams relevant to the current sub-event and composing a three-dimensional sound scene from these signals. An object-based scene model is authored, which references the audio capture streams and specifies the intended time-varying scene object positions.

During this process new audio source streams can also be created by combining, splitting, or modifying existing audio streams. For example a set of instrument spot microphones can be combined into a single instrument group stream.

The actors, a professional sound engineer and producer, can monitor the immersive audio scene on headphones and loudspeakers, using reference renderings. This monitoring process can also be used to preview the listening experience on different devices/systems.

This process creates the immersive audio stream of the sub-event for use in live editing of the event coverage (UC-3.2) and further distribution to audiences (UC-4 and UC-5). It is equivalent to the creation a channel-based (e.g. stereo) stream from audio sources for an event, by mixing them using a sound console and monitor loudspeakers, in a conventional production workflow.

During the immersive audio authoring process, the actors may refer to any existing capture metadata associated with the source streams, describing source location, content, microphone type etc., that was created during production preparation and capturing (see UC-1.2, UC-3.3, UC-2.3, UC-3.5, UC-3.6).

Metadata required for the semantic scene representation or that is needed to allow interactive experiences is also created and included in the scene model. The interactive elements of the scene rendering can be tested through the monitoring process. The actors may each monitor separate renderings of the scene to allow concurrent working on different elements of the experience.

Innovation

Representing an immersive audio scene using an object-based scene model

Immersive audio rendering of the scene model for headphones and loudspeakers

Streaming of an audio scene model in a production workflow

Concurrent monitoring of an audio scene using different renderings

Collaborative editing of an audio scene model

Real-time authoring and monitoring of interactive experiences

Dependencies

UC-2.3a UGC Capturing (during event)

UC-3.2 Live editing

UC-3.3 Data and metadata managementUC-3.5 Offline editing

UC-3.6 Professional capture

Actors

Professional sound engineer



Producer

Flow of events

For one sub-event/stage

1. Select audio source streams from list of those relevant and available, previewing audio signals if necessary

2. Specify/modify audio rendering configuration for monitoring

3. Arrange selected streams into a three-dimensional scene whilst monitoring the audio rendering

4. Create new audio streams from input streams if required or modify existing ones

5. Add/update scene and object metadata

6. Create/adjust interactivity metadata whilst monitoring the interactive audio experience

Inputs

[System] A set of available audio source streams and associated metadata streams

[System] Access to static metadata (sets/sub-events, schedules, playlists etc.)

[User] Audio stream selections made by the actors

[User] Selection of rendering configuration for monitoring made by the actors

[User] Creation/editing of spatial scene metadata for the audio objects in the audio scene made by the actors

[User] Creation/editing of semantic metadata for the audio scene and its audio objects made by the actors

[User] Creation/editing of interactivity metadata for the audio scene and its audio objects made by the actors

[User] Creation/editing of audio source streams made by actors

Outputs

[System] Stream of an immersive audio scene for the sub-event/stage (multichannel audio stream and scene model metadata stream)

[System] Renderings of the immersive audio scene for monitoring purposes

[User] Satisfied producer and sound engineer

Interfaces

Metadata model for an immersive and interactive audio scene of a sub-event/stage

Stream format definition combining/associating model and audio data

Metadata model describing the spatially outspread event (i.e., the “media and event scene graph”) and event schedule

User interaction for control and editing

Priority

High (8)



Use case diagram


Activity Chart



Figure 20: UC3.4 activity chart

Requirements






R3.11 Audio mixing


R3.13 Loudspeaker rendering



R3.14 Media coding

R3.15 Audio scene authoring interface




5.3.5 UC-3.5 Offline editing

Description

This use case describes the offline editing application. Crewmembers can browse through available content, edit this content add metadata (metadata enrichment). Afterwards, they can select it for on-demand. Available content can be UGC (including Moments), production content and so on. They can use social media and Moments information to consider which content to highlight.

Innovation

Integration of social media to highlight popular content

Automatic filtering of content (e.g. quality filters)

Add metadata (e.g. location) to content

Dependencies

UC-3.3 Data & metadata management



Actors

Crew members

Flow of events

1. Crew member opens post-production application

2. Available content is automatically filtered by quality, location and popularity (using social media and Moments information)

3. Crew member can add additional metadata or adjust existing metadata

4. Crew member selects content for on-demand

a. Professional production feed

b. Highlighted Moments by production on the Wall

c. Highlights for on-demand platform

Inputs

[System] Professional content

[System] UGC (including Moments)

[System] Production metadata

[System] Wall metadata

[System] Social media metadata

[System] Manual metadata

Outputs

[System] Professional production stream

[System] Moments on the Wall



[System] Highlights for on-demand platform

Interfaces Metadata, data

Priority

High (VRT) (4)

Use case diagram


Requirements











R3.11 Audio mixing





R3.14 Media coding



R4.4 Live playout service


Activity Chart

Figure 22: UC-3.5 activity chart

5.3.6 UC-3.6 Professional content capturing

Description

This use case describes the capturing of content using professional broadcast equipment and production staff.



Video is captured using HD broadcast cameras as well as advanced systems for immersive (pano/omni) video capture such as multi-camera arrays. Audio is captured using high quality microphones with existing techniques, as well as advanced systems for immersive (3D) audio capture such as spaced and coincident microphone arrays.

Microphones and cameras are deployed as specified in the pre-production phase, and this information is available to the production team through the ICoSOLE production application.

Calibration of the microphone and camera systems is performed using conventional procedures and new efficient methods.

Captured content is streamed in real-time into the ICoSOLE system for live production and storage. Other production team members on the ICoSOLE system can then immediately access this content (for monitoring/editing/broadcasting).

The ICoSOLE production application can be used to specify semantic and technical metadata describing the professional content sources. This can be accessed from a tablet device or a production workstation. The streams can be linked with a specific event (stage) and device location, media format, recording parameters, device description, content source and other parameters can be specified to assist with editing/mixing.

Any pre-processing to make the streams useable for mixing/editing, is configured within the ICoSOLE production system (e.g. combining camera/mic arrays into a single immersive stream). The parameters of this processing can be configured using the ICoSOLE production application and the relevant captured content can be routed to this process.

Talk-back communication is available between the capture operators and the director and producer.

Innovation

Real-time IP-based ingest and streaming of multiple audio and video signals

Real-time meta-data ingest and stream association using remote service

Real-time processing of captured signals in the IP-based system.

Video capture using panoramic and omnidirectional systems

Audio capture using novel microphone arrays

Efficient calibration tools for camera and microphone setup

Dependencies


UC-1.3 Managing event calendar and locations

UC-1.6 Site survey for professional capture

Actors

In the field

o Camera operator

o Sound operator

In production centre

o Sound engineer

o Video engineer

o Director

o Producer

Flow of events

1. Camera and sound operators arrive at event location and rig equipment as planned

2. Cameras and microphones are connected to the ICoSOLE system

3. Technical metadata associated with the devices is added/edited

4. Calibration of the devices is performed

a. Engineers, producer and director remotely monitor devices



b. Any required processing of devices is configured in the ICoSOLE system

5. Semantic metadata is added describing signal contents

6. Capture of input signals is activated in ICoSOLE system

7. Performance/event begins

8. Devices and metadata are adjusted if required during capture

Inputs

[System] Event timetable and production planning on ICoSOLE platform

[System + User] Microphone and camera equipment

Outputs

[System] A set of audio and video signals ready for mixing/editing into streams for broadcast

[System] Associated meta-data describing these sources

Interfaces

Ingest and routing (identify input sources)

Source metadata editing (associated and update metadata with sources)

Processor control

Preview monitoring of capture signals

Calibration tools.

Priority

High (9)



Use case diagram




Activity Diagram

Figure 24: UC3.6 activity chart



Requirements



R2.1 Calibration of used technical instruments


R2.3 Capturing audio



R3.14 Media coding




5.3.7 UC-3.7 Advanced video authoring and rendering

Description

This use case is concerned creating the output video scene, i.e. selecting the streams to be rendered together or separately, finally adjusting their alignment for rendering, and selecting possible in/out points and transition options.

The input is the capture scene after automatic filtering and manual updates in the metadata management UC, and the output are the new streams and associated rendering instructions and metadata. This includes selecting viewports from panoramic cameras to feed traditional video streams.

The output streams are also assigned to (a single or multiple) delivery target.

Innovation

Integrated selection of panoramic and conventional video streams

Transitions between spatially aligned video streams

Dependencies

UC-2.3 UGC Capturing (during event)

UC-3.2 Live editing

UC-3.3 Data and metadata management



UC-3.6 Professional capture

Actors

Video engineer

Producer

Flow of events

For one sub-event/stage

1. Select the video streams to be considered from the streams covering the area

2. Choose overlapping streams to be rendered together (e.g. from panoramic camera)

3. Define entry/exit points from some streams

4. Update final spatial arrangement of streams

5. Select from transition options and apply the transitions

6. Assign streams to specific delivery channels



Inputs

[System] A set of available video source streams and associated metadata streams

[System] Access to static metadata (sets/sub-events, schedules, playlists etc.)

[User] Video stream selections made by the actors

[User] Selection of streams and rendering options made by user

[User] Update of stream/transition metadata

Outputs

[System] Set of video streams and associated rendering options

[System] Assignment of streams to specific delivery channels

Interfaces

Capture scene representation on the input side

Video production scene representation of the output side

Metadata streams, in particular location, results from automatic analysis/filtering

Priority

High (JRS, VRT) (8)

Use case diagram




Requirements



R3.1 Automatic alignment of pictures taken in site survey





R3.9 Media coding

R3.16 Video scene authoring interface




5.4 UC-4 Playback on TV

5.4.1 UC-4.1 Live playback on TV

Description

This use case details the scenario for displaying live content on the primary home TV screen. The principal focus is on efficient distribution of high quality multimedia streams as well as limited interactions (as dictated by the output device). The media distribution will preferably be MPEG-DASH-based. The interaction options must at least allow consumers to switch between (the live streams of) the constituting, spatially outspread sub-events that are taking place in parallel. Combining multiple views on the same content is controlled through split screen functionality, including various types of content (e.g. traditional video, omni-directional or panoramic).

Given the availability of more high-end sound output devices, a more convincing and high quality experience is targeted when compared to the mobile scenario (see UC-5.1). Therefore, this use case targets (3D) audio rendering, while at the same time providing advanced audio management functionalities like crowd noise control, commentary adjustment, the ability to adopt alternative positions in the audience, and the selection of audio objects.

As most consumers still either use the TV’s internal stereo loudspeakers, or possibly a 5.1 surround setup using their hi-fi, we must assume that the speaker layouts under test will be a minimum stereo and 5.1 surround. Therefore high quality reproduction must be capable of these two speaker layouts. If a full 3D speaker layout is available then the renderer within the TV/STB/amplifier should be able to provide suitable full 3D rendering from the broadcast audio stream.

Please notice that the adaptive features of DASH are left out of the equation in this use case. As this usage scenario involves delivery to residential end-points, it is reasonable to assume that the consumer has a fixed and therefore relatively stable network connection to the content producer/distributor, and that this connection is sufficiently capacitated in terms of network bandwidth to warrant the streaming of the audiovisual content in maximal quality.

Innovation

Design of an efficient distribution method for delivering the same content simultaneously to large numbers of viewers. Due to the poor scalability properties of unicast traffic, the content dissemination scheme will need to technologically surpass one-to-one connections. Also, by adopting efficient caching schemes located close to the end-user and which do not interfere with the chosen streaming method, the delivery efficiency might be improved.

Fast and uninterrupted switching between views or displaying them simultaneously in either a side-by-side or picture-in-picture fashion (comparable to the issue of zapping speed in TV broadcasts).



Providing efficient, user-friendly and potentially immersive means that simulate physical navigation between spatially outspread concurrent sub-events. This will involve the construction of an “event and media scene graph” that is correctly staged in the physical environment in which the composite event takes place (e.g., the hosting city in case of a municipal music festival).

Controlling the limited interaction facilities through basic remotes and other off-the-shelf input devices suitable for the TV set.

Spatial (3D) audio rendering created from the format agnostic representation of the large-scale event.

Suitable audio rendering to stereo and 5.1 speaker layouts from 3D object-based material.

3D audio integration to MPEG-DASH.

Dependencies

Receives input from

UC-3.2: Live editing

UC-3.3: Data and metadata management

Is linked to

UC-4.2: On-demand playback on TV

UC-5.1: Live playback on mobile

UC-5.2a: On-demand playback on mobile

Actors

Consumer

Producer

Flow of events

1. The producer(s) continuously prepare(s) the live broadcast signal for each of the concurrent sub-events.

2. (A subset of) the live broadcast streams are delivered in real-time to the consumer using DASH.

3. The consumer accesses the live broadcast(s) on his TV screen by tuning in to the appropriate channel.

4. The consumer interacts with the live broadcast(s) via a simple, intuitive Graphical User Interface (GUI) which he controls via an off-the-shelf input device that is suitable for the TV set (e.g., a remote control).

a. The GUI allows the consumer to switch between (the live streams that correspond with) the different locations/stages that constitute the spatially outspread event.

b. The GUI allows the consumer to install multiple views on the same content, for example through split screen functionality.

c. The GUI allows the consumer to control the audio characteristics such as crowd and commentary noise, and listening position.

5. The consumer ends his session by switching to another TV channel or by shutting down his TV set.

Inputs

[System] The network streams representing the live broadcast(s).

[User] User input in order to control the media presentation (e.g., content/stage selection, view and perspective selection, navigation between spatially dispersed yet temporally parallel sub-events).

[System] Metadata about the event that is being covered by the live broadcast(s). For example, in case a representation of the “media and event scene graph” would be available at client side, it would become feasible to grant the consumer an impression of the physical space in which



the spatially outspread event takes place, to support event exploration, and to enable the consumer to traverse the participating sites in the composite event.

Outputs

[User] A hopefully satisfying and immersive consumer experience in which the end-user is afforded a comprehensive impression of the spatially dispersed event in real time (i.e., as it is unwinding at this very moment) and at a high audiovisual fidelity.

Interfaces

The live “broadcast” signal as authored by the ICoSOLE producer(s).

Metadata, including (but potentially not limited to) a model of the spatially outspread event (i.e., the “media and event scene graph”), and an event schedule including biographical information of the involved performers.

Priority

High (BIT, iMinds) (5)



Use case diagram


Requirements




R3.14 Media coding



R5.3 Playout client for live playback on TV

R5.4 Content pre-caching to improve broadcast channel zapping on TV



R5.6 Control of TV stream selection

5.4.2 UC-4.2 On-demand playback on TV

Description

This use case details the scenario for displaying multimedia content on the primary home TV screen. The principal focus is on efficient, DASH-based distribution of high quality multimedia streams as well as limited interactions (as dictated by the output device). The interaction options must at least allow consumers to step back and forth (both on the timeline and spatially) and review multiple clips of dedicated (scenes) from different angles to get an immersive impression of the entire event, almost as if the viewer was on the spot. Possibility to watch a summary (composed by the production crew) or browse and select all available content (sorted by time, location and performing artist) is provided. Navigate through all different camera and audio streams, provides the ability to experience the festival again with a complete different choice of program.

Combining multiple views on the same content is controlled through split screen functionality, including various types of content (e.g. traditional video, omni-directional or panoramic).

Given the availability of more high-end sound output devices, a more convincing and high quality experience is targeted when compared to the mobile scenario (see UC-5.2). Therefore, this use case targets (3D) audio rendering, while at the same time providing advanced audio management functionalities like crowd noise control, commentary adjustment, the ability to adopt alternative positions in the audience, and the selection of audio objects.

Innovation

UGC synchronization to professional content in an interactive and immersive way

Fast and uninterrupted switching between views or displaying them simultaneously in either a side-by-side or picture-in-picture fashion (comparable to the issue of zapping speed in TV broadcasts)

Ensuring smooth content consumption via adaptive delivery (leveraging the MPEG-DASH client library libdash)

3D loudspeaker audio rendering created from the format agnostic representation of the large-scale event

3D audio integration to MPEG-DASH

Interactive functionality to explore the different versions of the content (i.e. traditional, panoramic, omni-directional and free-viewpoint) via an intuitive user interface

Dependencies

Receives input from

UC-2.3a: UGC Capturing



Is linked to

UC-4.1: Live playback on TV

UC-5.2a: On-demand playback on mobile

Actors

Consumer

Flow of events

1. Multimedia content is delivered to the consumer using DASH

2. The consumer accesses the broadcast(s) on his TV screen by tuning in to the appropriate channel



3. The consumer interacts with the platform via a simple, intuitive Graphical User Interface (GUI) which he controls via an off-the-shelf input device that is suitable for the TV set (e.g., a remote control)

a. The GUI allows the consumer to switch between the different locations/stages that constitute the spatially outspread event

b. The GUI allows the consumer to install multiple views on the same content, for example through split screen functionality

c. The GUI allows the consumer to step back and forth on the timeline

d. The GUI allows the consumer to install multiple views on the same content

e. The GUI allows the consumer to select a POI/lock the viewing location (e.g. to his/her favorite rider)

f. The GUI allows the consumer to select different audio playback styles and to perform advanced audio management (e.g., control the level of crowd noise)

4. The consumer is able to watch a summary composed by the production crew

5. The consumer ends his/her session

Inputs

[System] The network streams representing the multimedia content

[User] User input in order to control the media presentation (e.g., content/stage selection, view and perspective selection, navigation between spatially dispersed yet temporally parallel sub-events)

[System] Metadata about the event that is being covered

Outputs

[User] Satisfied consumer

Interfaces

DASH signal authored by the ICoSOLE producer(s)

Metadata, including (but potentially not limited to) a model of the spatially outspread event (i.e., the “media and event scene graph”), and an event schedule

Priority

High (BIT) (3)



Use case diagram


Requirements



R3.13 Loudspeaker rendering

R3.14 Media coding



R5.4 Content pre-caching to improve broadcast channel zapping on TV




5.5 UC-5 Playback on mobile device

5.5.1 UC-5.1 Live playback on mobile device

Description

This use case deals with the live playback of multimedia content on mobile devices. It results in an interactive live stream adapted to device and bandwidth conditions.

The live stream of an event can be viewed via the ICoSOLE web app (adapted to bandwidth conditions and device specifications), with the ability to follow the happenings based on (from the production crew) selected scenes or “locked” to a specific location (or athlete in case of a sport event) by selection. Further, controlling audio functionalities like crowd noise and commentary adjustment and choosing an alternative position in the audience with binaural audio experience is provided. Dynamic binaural audio streams are provided with the use of head tracking, affording an immersive headphone sound experience.

Innovation


Ensuring smooth content consumption via adaptive delivery (based on a client application for smartphones leveraging the MPEG-DASH client library libdash)

Binaural audio rendering created from the format agnostic representation of the large-scale event


Interactive functionality to explore the different versions of the content (i.e. traditional, panoramic, omni-directional and free-viewpoint) and different audio perspectives (i.e. using head tracking and foreground/background mix) via an intuitive user interface

Dependencies

Receives input from

UC-2.3a: UCG Capturing



Is linked to

UC-4.1: Live playback on TV

UC-5.2a: On-demand playback on mobile device

Actors

Consumer

Flow of events

1. (A subset of) the live broadcast streams are delivered in real-time to the consumer using DASH

2. The consumer accesses the live broadcast(s) on his/her mobile device

3. The consumer interacts with the live broadcast(s) via a simple, intuitive Graphical User Interface (GUI)

a. The GUI allows the consumer to switch between (the live streams that correspond with) the different locations/stages that constitute the spatially outspread event

b. The GUI allows the consumer to install multiple views on the same content

c. The GUI allows the consumer to select a favourite artist/athlete




Inputs

[System] The network streams representing the live broadcast(s)

[User] User input in order to control the media presentation (e.g., content/stage selection, view and perspective selection, navigation between spatially dispersed yet temporally parallel sub-events)

[User] Head tracking device input for binaural rendering

[System] Metadata about the event that is being covered by the live broadcast(s)

Outputs


Interfaces

Live DASH signal authored by the ICoSOLE producer(s)


Priority

High (7)

Steps (high level)

Figure 28: UC-5.1 high level diagram

Use case diagram




Activity Diagram


Requirements






R3.14 Media coding



R5.1 Playout platform for mobile users

5.5.2 UC-5.2a On-demand playback on mobile device

Description

This use case deals with the playback of multimedia content on mobile devices. It results in an interactive experience for consumers adapted to device and bandwidth conditions.

Provides the ability to re-experience the event on-demand with the ability to step back and forth (both on the timeline and spatially) and review multiple clips of dedicated (scenes) from different angles to get an immersive impression of the entire event, almost as if the viewer was on the spot. The possibility to watch a summary (composed by the production crew), or browse and select all available content (sorted by time, location and performing artist). Navigate through all different camera and audio streams, provides the ability to experience the festival again with a complete different choice of program. Further, controlling audio functionalities like crowd noise and commentary adjustment and choosing an alternative position in the audience with binaural audio experience is provided. Dynamic binaural audio streams are provided with the use of head tracking, affording an immersive headphone sound experience.

Innovation


Ensuring smooth content consumption via adaptive delivery (based on a client application for smartphones leveraging the MPEG-DASH client library libdash)

Binaural audio rendering created from the format agnostic representation of the large-scale event



Dependencies

Receives input from:

UC-2.3a UGC Capturing

UC-3.2 Live editing

UC-3.3 Data and metadata management.

Is related to:

UC-4.1 On-demand playback on TV.

UC-5.1 Live playback on mobile device.

Actors

Consumer

Flow of events

1. Multimedia content is delivered to the consumer using DASH 2. The consumer accesses the playout platform on his/her mobile device 3. The consumer interacts with the platform via a simple, intuitive Graphical User Interface (GUI)

a. The GUI allows the consumer to switch between the different locations/stages that constitute the spatially outspread event

b. The GUI allows the consumer to step back and forth on the timeline



c. The GUI allows the consumer to install multiple views on the same content d. The GUI allows the consumer to select a POI/lock the viewing location (e.g. to his/her

favourite rider) 4. The consumer is able to watch a summary composed by the production crew 5. The consumer ends his/her session

Inputs


[User] User input in order to control the media presentation (e.g., content/stage selection, view and perspective selection, navigation between spatially dispersed yet temporally parallel sub-events, …)


[System] Metadata about the event that is being covered

Outputs


Interfaces



Priority

High (8)

Use case diagram





Activity Diagram

Figure 33: UC-5.2a Activity diagram

Requirements

R1.2 Reading & storing event calendar




R3.14 Media coding



5.5.3 UC-5.2b The Wall of Moments

Description

This use case describes a mobile feature called ‘The Wall of Moments’, in which a remote user can have a unique festival experience by using ‘Moments’ captured by peers attending the festival. He pitches in to the concert via a Moment and can enjoy a high quality A/V stream from there on, as Moments are precisely synchronized with the production feed. This way, the user still has an immersive experience by having direct social contact with the crowd (more importantly, his/her friends/peers).

The Wall of Moments is a mosaic representing the most popular moments users shared (e.g. via Facebook, Twitter), moments the production highlighted, live production feeds and a virtual map. It acts as an online interactive portal to the festival for users at home and on the way.

The Wall contains one column of dedicated tiles, whereas the other tiles’ content varies based on own interests, friends’ activity and suggestions, production highlights and so on. The dedicated column contains a tile for the live production feed (Now), a Last Viewed tile and a tile to view a virtual map of the festival, see UC-5.2c. When the Live tile is tapped, the production feed of the current concert starts playing. Users can navigate to other stages/camera angles by swiping. All other tiles of the Wall play a muted video feed of an interesting Moment. When tapped, the Moment enlarges to fit the screen and plays unmuted. When available, a production A/V feed plays along picture-in-picture. When the Moment finishes playing, the picture-in-picture A/V feed takes over the screen and an annotated Timeline appears at the bottom.

The start of the Timeline marks the beginning of the current concert and end marks ‘Now’. Interesting event Moments (shared by friends or production) are shown on the Timeline and allow users to navigate through a concert socially, instead of a linear view. Users are given the choice to continue watching from this moment, from the beginning of this song or from the beginning of the concert. If users want to sit back and relax, they can mirror the production feed to their TV set (AirPlay). Viewed Moments are added to a Viewed Moments list (which can later be used to create My Own Festival Experience to share with friends), can be manually appended to Favourites and can be shared (again). Lastly, users can also go straight back to the Wall, where the watched tile is replaced by a new one and replaces the dedicated Last Viewed tile.

The Wall of Moments can also be used by the professional production crew to feature on the festival website or during concert breaks on the video walls.

Innovation

Integration of social media to highlight popular content

Automatic filtering of content (e.g. quality filters)

Add metadata (e.g. location) to content

Dependencies

UC-3.3: Data & metadata management



UC-5.1: Live playback on mobile device

Actors

Remote users (e.g. at home, on the way, on the festival camping)

Festival production (e.g. director, editorial office)



Flow of events

1. A user at home or on the way opens the mobile application and gets an overview of current Moments (The Wall of Moments)

2. The user taps a Moment, which stretches to fit the screen and starts playing

3. The production feed is playing along PIP, synchronized with the Moment

4. When the Moments end or when the user taps the PIP, the production feed continues playing full screen

5. The user can use the annotated timeline on the top to navigate to other interesting Moments in this concert or can go back to The Wall, where the played Moment is moved to the Last Played tile and a new Moment appears

Inputs

[System] Professional content & metadata

[System] UGC (including Moments)

[System] Social media metadata

[System] Manual metadata

Outputs

[User] The Wall of Moments

Interfaces

Metadata, data

Priority

High (VRT) (5)



Use case diagram

Figure 34: UC-5.2b diagram

Requirements






R3.14 Media coding


R5.2 Interactive Wall

R5.7 Moment selection

Activity Chart



Figure 35: UC-5.2b activity chart

5.5.4 UC-5.2c Venue Explorer

Description

This use case describes a mobile/tablet application called ‘Venue Explorer’, in which a remote user can navigate the event coverage using an interactive map or panoramic video of the entire event. The user can pan/zoom around this global view of the scene to explore the areas of most interest to them. Available A/V content of sub-events and Moments is located at the correct position within the scene and indicated to the user for selection. Interactive graphical overlays can be presented as well as A/V previews of the content. A timeline control is also available to the user, to display the relevant content for a particular time during the event.

The user can select a piece of produced A/V content for a sub-event from the Venue Explorer interface for full-screen viewing. The user can return to the Venue Explorer navigation view at any time.

During full-screen viewing similar functions to UC-5.2a are available. Interactive controls like crowd noise and commentary adjustment are available. Different A/V perspectives on the sub-event can also



be accessed when available and dynamic binaural audio streams are provided with the use of head tracking, affording an immersive headphone sound experience.

Innovation

Intuitive interactive spatio-temporal navigation of many content feeds for a large event

Interactive overlays on live video


Ensuring smooth content consumption via adaptive delivery (including pan/zoom in panoramic background video)


Immersive binaural audio rendering with head tracking on a mobile device


Dependencies

UC-3.3: Data & metadata management



UC-5.1: Live playback on mobile device

UC-5.2a: On-demand playback on mobile device

UC-5.2b: The Wall of Moments

Actors

Consumer

Flow of events

1. The consumer accesses the playout client on his/her mobile device and selects the Venue Explorer interface mode

2. A scene representation of available content is delivered to playout client including the spatiotemporal extent of each content item

3. Multimedia content is delivered to the consumer using DASH 4. The consumer interacts with the platform via the Venue Explorer interface

a. The GUI presents the consumer with a global map-view of the event, indicating positions of available content items.

b. The GUI allows the consumer to move back and forth on the timeline c. The GUI allows the consumer to pan/zoom around the global scene view d. The GUI allows the consumer to preview available content, and receive automatic picture-

in-picture previews based on current viewport. e. The GUI allows the consumer to turn on and interact with overlay graphics

5. The consumer is able to select and view available content items in full-screen mode a. The consumer orientation is tracked to provide dynamic binaural audio b. The GUI allows the consumer to adjust a POI for certain content items


Inputs

[System] Metadata about the event that is being covered (inc. scene representation)




[User] User input in order to control the media presentation (e.g. content/stage selection, view and perspective selection, navigation between spatially dispersed yet temporally parallel sub-events)


[System] Panoramic global view of the scene (video/image)

[System] Description of overlay graphics to present

Outputs


Interfaces


Metadata, including (but potentially not limited to) a model of the spatially outspread event (i.e., the “media and event scene graph”), an event schedule, and supplementary data for graphical overlays

Priority

High (BBC) (6)

Use case diagram

Figure 36: UC-5.2c diagram

Requirements







R3.14 Media coding



R5.5 Venue explorer



6 Requirements

From the use case descriptions, a list of requirements of the ICoSOLE system has been generated. These requirements can be used to inform the design and implementation of the ICoSOLE system.

6.1 Service Requirements

This section lists required services derived from the use case descriptions of the previous section, such a service will provide certain specific functionality required of the ICoSOLE system. Detailed definitions of these service requirements are provided in section 7.

The following categories are used for these service requirements:

Metadata: Services dealing with metadata rather than media data

Capture: Services dealing with capture of media data into the ICoSOLE system

Processing: Services dealing with processing of captured media data

Distribution: Services concerning distribution of data around the ICoSOLE system and to audiences

Audience: Services provided to the audience

Table 2: List of service requirements

ID Requirement

name

Use cases

Type

Contributing partners

1.1 1.2 1.3 1.4 1.5 1.6 2.1 2.2 2.3a2.3b

3.1 3.2 3.3 3.4 3.5 3.6 3.7 4.1 4.2 5.1 5.2a 5.2b 5.2c JRS DTO VRT iMinds BIT BBC TaW


x x x x x x x Metadata x

R1.2 Reading & storing event calendar metadata x x x x x x x x x x x x x x x x x Metadata x

R1.3

Reading & storing technical equipment metadata

x x x x x Metadata x x

R1.4 Reading & storing production crew information x x x x x Metadata x

R1.5

Reading & storing UGC capture requests and intentions

x x x x x Metadata x



R1.6 Reading & storing 3D model of event location x x x x x x x x x Metadata x

R1.7 Convert metadata to a common standard x Metadata x

R1.8

Reading & storing UGC metadata (comments, location, sensors,…)

x x x x x x Metadata x

R1.9 Messaging system for users & producers x x Metadata x x

R1.10 Social media integration x x x x Metadata x

R2.1 Calibration of technical equipment

x x Capture x x x

R2.2 Capturing video x x x Capture x x

R2.3 Capturing audio x Capture x x

R2.4 Upload service x x Capture x

R3.1

Automatic alignment of pictures taken on site survey

x x x Processing x

R3.2 Automatic filtering of streams x x x x Processing/

Metadata x

R3.3 Broadcast graphics service x x Processing x

R3.4 Timeline alignment of content x x x x x x Processing x x

R3.5 Live editing service x x Processing x

R3.6 Editing scene representation x x x Processing/

Metadata x

R3.7 Automatic extraction of metadata x x Processing/

Metadata x



R3.8 Semi‐automatic linking of content with metadata x x x x Processing/

Metadata x

R3.9 Audio processing x x x x Processing x x

R3.10 Audio analysis x x x x Processing x x

R3.11 Audio mixing x x x Processing X

R3.12 Headphone sound rendering x x x x x x Processing X

R3.13 Loudspeaker sound rendering x x x x x Processing x

R3.14 Media (en‐/de‐/trans‐) coding x x x x x x x x x x x x x Processing x


x Processing/metadata x


x Processing/metadata x x

R4.1 Synchronisation signal distribution x x x x x Distribution x


x x x x x x x x Distribution x

R4.3 Production data storage x x x x x x x x Distribution x

R4.4 Live playout service x x Distribution x

R4.5

Scalable IP‐based distribution scheme for live playback

x x x Distribution x

R4.6 Adaptive bit‐rate media delivery x x x x x x x x Distribution x x

R5.1 Playout platform for mobile users x x x Audience x x

R5.2 Interactive Wall x Audience x


x Audience x x



R5.4

Content pre‐caching to improve broadcast channel zapping on TV

x Audience/ Distribution x

R5.5 Venue explorer x Audience x

R5.6 Control of TV stream selection x x Audience x

R5.7 Moment selection x Audience x



6.2 User Interfaces

The following services require user interfaces:




R3.8 Semi-automatic linking of content with meta-data




R5.2 Interactive Wall


R5.5 Venue Explorer


In addition, the following user interfaces need to be considered:

mobile UI for UGC capture

UI for managing users, capture requests, incentives, etc

6.3 Non-Functional Requirements

This section lists non-function requirements of the ICoSOLE system, holistic properties that describe the behaviour of the system.

Scalable - The ICoSOLE system should be scalable, capable of adapting to events and audiences of the variety of sizes that might be encountered in broadcasting.

Adaptive – The media output of the system should be able to adapt to different scenarios/platforms.

Configurable - The ICoSOLE system should be configurable, comprising of a set of tools that can be applied in a variety of ways to suit any particular event production.

High-Quality – The ICoSOLE system should produce media output to a high quality level, acceptable to broadcasters and their audiences.

Distributed – The ICoSOLE system should be capable of being distributed over a wide area, allowing for use in production of large events.



7 Requirements Specifications

This section provides detailed specifications of each required service. It includes the inputs and outputs, as well as required resources and dependencies.

7.1 Metadata Services

7.1.1 Reading and storing administrative meta-data

Contributing partners VRT Applies to use cases 1.1, 1.2, 1.5, 2.1, 2.2, 5.2b Type Metadata

Summary Brief overview of the requirements findings This service provides access to administrative metadata via a common (REST) interface. Administrative metadata can be read, added, updated and removed. This service is protected from unauthorized access and allows for users to authenticate and start a session. Scope & purpose What is to be achieved with this service? What is needed? Making administrative metadata persistent to be able to use it wherever necessary via a common (protected) REST interface. Inputs List of the required and optional input for the service User authentication, administrative metadata, e.g. user information Outputs List of the output of the service Administrative metadata Flow Description of the event flow when the service is activated - User authenticates himself using a username and a password - On successful authentication, the user is granted access to the administrative metadata REST service - User can read, add, update and remove administrative metadata via REST Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST using JSON definition language Required resources Description of (external) resources involved / Dependencies Software, hardware and platform dependencies Database (e.g. MySQL), webserver (e.g. Apache), PHP or third-party (Firebase) Related information Documents, links to the service http://www.mysql.com http://php.net http://www.apache.org https://www.firebase.com

7.1.2 Reading and storing event calendar meta-data

Contributing partners VRT Applies to use cases 1.2, 1.3, 1.5, 1.6, 3.2, 3.3, 3.4, 3.6, 5.1, 5.2a, 5.2b Type Metadata

Summary Brief overview of the requirements findings This service provides access to calendar data via a common (REST) interface. Calendar metadata can



be read, added, updated and removed. This service can import calendar data from third-party resources, using linked open data and is protected from unauthorized access (e.g. public key authentication). Scope & purpose What is to be achieved with this service? What is needed? Making calendar metadata persistent to be able to use it wherever necessary via a common (protected) REST interface. Integrating ICoSOLE specific and third-party calendar information. Inputs List of the required and optional input for the service Authentication information (e.g. public key), calendar metadata (ICoSOLE and third-party) Outputs List of the output of the service Calendar metadata Flow Description of the event flow when the service is activated - User authenticates himself (e.g. public key) - On successful authentication, the user is granted access to the administrative metadata REST service- User can add, read, update and remove calendar metadata via REST - User can modify third-party calendar resources (linked open data) Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema) REST using JSON definition language Required resources Description of (external) resources involved Third-party calendar resources (linked open data) Dependencies Software, hardware and platform dependencies Database (e.g. MySQL), webserver (e.g. Apache), PHP or third-party (Firebase) Related information Documents, links to the service http://www.mysql.com http://php.net http://www.apache.org https://www.firebase.com

7.1.3 Reading and storing technical equipment meta-data

Contributing partners BBC, BIT Applies to use cases 1.2, 1.6, 3.3, 3.6 Type Metadata

Summary Brief overview of the requirements findings All the technical equipment, including cameras and microphones will need their technical metadata read and stored. This will be done before the event starts and would be part of the setting up process by the producer and their team. Scope & purpose What is to be achieved with this service? What is needed? To ensure all the capture devices and other relevant technical equipment has enough information about them recorded for the purposes of the shoot. This information would be stored as metadata that will be managed and stored with the scene representation metadata. While much of the metadata will have to be generated manually, there is scope for automation for some aspects of it. Inputs List of the required and optional input for the service Equipment details Static parameter from capture devices Location measurements Outputs



List of the output of the service Position metadata for devices Capture device parameter metadata Flow Description of the event flow when the service is activated 1. Technical information about each device is noted. 2. Location scout records 3D locations of all the devices. 3. Manually obtained information is entered via metadata logging system. 4. Parameters for each are converted to capture metadata Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)The service shall support at least one of the following interfaces: * RESTful interface using JSON * Tablet based GUI for easy metadata entry in the field. Required resources Description of (external) resources involved * Production team for entering metadata. Dependencies Software, hardware and platform dependencies * Tablets (or similar hand-held devices) for metadata entry. Related information Documents, links to the service

7.1.4 Reading and storing production crew information

Contributing partners VRT Applies to use cases 1.2, 1.3, 1.5, 3.3 Type Metadata

Summary Brief overview of the requirements findings This service provides access to production crew metadata via a common (REST) interface. Production crew metadata can be read, added, updated and removed. This service can incorporate production crew metadata from an external production application (e.g. using linked open data) and is protected from unauthorized access (e.g. public key authentication). Scope & purpose What is to be achieved with this service? What is needed? Making production crew metadata persistent to be able to use it wherever necessary via a common (protected) REST interface. Integrating production crew metadata from ICoSOLE and external applications. Inputs List of the required and optional input for the service Authentication information (e.g. public key), production crew metadata (ICoSOLE and external production applications). Outputs List of the output of the service Production crew metadata Flow Description of the event flow when the service is activated - User authenticates himself (e.g. public key) - On successful authentication, the user is granted access to the production crew metadata REST service - User can add, read, update and remove production crew metadata via REST - User can modify resources from external production applications (e.g. linked open data) Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST using JSON definition language



Required resources Description of (external) resources involved External production application crew information (e.g. using linked open data) Dependencies Software, hardware and platform dependencies Database (e.g. MySQL), webserver (e.g. Apache), PHP or third-party (Firebase) Related information Documents, links to the service http://www.mysql.com http://php.net http://www.apache.org https://www.firebase.com

7.1.5 Reading and storing UGC capture requests and intentions

Contributing partners VRT Applies to use cases 1.5, 2.2, 2.3a, 3.3 Type Metadata

Summary Brief overview of the requirements findings This service provides access to UGC requests and intentions via a common (REST) interface. UGC requests and intentions can be read, added, updated and removed. This service is protected from unauthorized access and allows for users to authenticate and start a session. A producer can issue a request for UGC via an app, which uses this REST interface to store the request. A user can contribute to a request by adding an intention for the request in a client app, which uses the same REST interface to save this intention. Scope & purpose What is to be achieved with this service? What is needed? Making UGC requests and intentions persistent to be able to use it wherever necessary via a common (protected) REST interface. This metadata can be used by the production team to map which areas are well covered (by UGC). This way, UGC can be steered and used more efficiently. Inputs List of the required and optional input for the service User authentication, administrative metadata (user information), UGC requests (from a production crew member) and intentions (from a festival participant) Outputs List of the output of the service UGC requests (what, where and when should be covered by festival participants) and intentions (who will cover it) Flow Description of the event flow when the service is activated - User authenticates himself using a public key or username and password (link do administrative metadata) - On successful authentication, the user is granted access to the UGC requests and intentions REST service - User can read, add, update and remove UGC requests and intentions via REST: - Requests are added by production crew members (e.g. stages not covered by a professional crew) - Festival participants contribute to these requests by saving capture intentions Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST using JSON definition language GUI for easy metadata (requests and intentions) entry Required resources Description of (external) resources involved Administrative metadata service Metadata entered by user Dependencies



Software, hardware and platform dependencies App for production crew members App for festival participants Database (e.g. MySQL), webserver (e.g. Apache), PHP or third-party (Firebase) Related information Documents, links to the service http://www.mysql.com http://php.net http://www.apache.org https://www.firebase.com

7.1.6 Reading and storing 3D model of event location

Contributing partners JRS Applies to use cases 1.6, 3.1, 3.7, 4.1, 4.2, 5.1, 5.2b, 5.2c Type Metadata

Summary Brief overview of the requirements findings Save and retrieve 3D model information of the event location. Scope & purpose What is to be achieved with this service? What is needed? Save 3D information gathered from site survey and measurements, and allow retrieval of the entire or partial model for use in scene definition and playout. Inputs List of the required and optional input for the service - a 3D model to be stored - a request to return the entire model of the location or a part of the model covering a specific area Outputs List of the output of the service - a 3D model of the requested (part) of the set Flow Description of the event flow when the service is activated - reading -- provide metadata model -- store model to local storage - reading -- specify area to be retrieved (as identifier of set/subset) -- get model from local storage -- return model (presumably as one blob, not streamed) Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST using XML data representation for requests, and a standard 3D representation (eg COLLADA) for the model Required resources Description of (external) resources involved Dependencies Software, hardware and platform dependencies Related information Documents, links to the service

7.1.7 Converting metadata to a common standard

Contributing partners JRS Applies to use cases 3.3 Type Metadata

Summary



Brief overview of the requirements findings Convert incoming metadata from their native representation to the agreed representation in the ICoSOLE system for this type of metadata. Scope & purpose What is to be achieved with this service? What is needed? Ensure interoperability of metadata throughout the ICoSOLE system by having a defined representation of metadata in each of the streams. Inputs List of the required and optional input for the service - One or more metadata streams (in their native format) Outputs List of the output of the service - One or more metadata stream(s) in interoperable format Flow Description of the event flow when the service is activated - Read a metadata sample - perform preconfigured configuration - output converted metadata sample Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST using XML data representation Required resources Description of (external) resources involved - Metadata mapping definitions Dependencies Software, hardware and platform dependencies Related information Documents, links to the service

7.1.8 Reading and storing UGC metadata

Contributing partners BIT Applies to use cases 1.1, 2.3a, 3.2 Type Metadata

Summary Brief overview of the requirements findings All user devices (e.g. smartphones with cameras) will need their metadata read and stored. This will be done before the user starts recoding, periodically during the shoot (by the application), as well as at arbitrary points in time, when the user adds additional information (e.g. comments). Scope & purpose What is to be achieved with this service? What is needed? To ensure all captured UGC has enough information about them recorded for the purposes of the shoot. This information would be stored as metadata. Much of the metadata will be generated automatically; some metadata will have to be generated manually as well. Inputs List of the required and optional input for the service Sensor (e.g. location, light conditions...) data from capturing device Manually added information from user Outputs List of the output of the service Position metadata from the device Various metadata linked to time and quality of the recoding depending on the capabilities of the device Metadata manually added by the user Flow Description of the event flow when the service is activated



1. Time and location information is automatically generated by the smartphone application at start-up 2. Manually obtained information is entered via the smartphone application 3. Sensor and location information is automatically stored during the capturing 4. Technical and non-technical metadata is handed over to producers Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)Interface using JSON GUI for easy metadata entry in the field. Required resources Description of (external) resources involved User entering information Dependencies Software, hardware and platform dependencies Smartphone application for content capturing, automatic sensor reading (and storing) and possibility to add information manually Automatic generated metadata is highly dependent on the device (capabilities) Related information Documents, links to the service

7.1.9 Messaging system for users and producers

Contributing partners BIT, VRT Applies to use cases 2.2, 2.3a Type Metadata

Summary Brief overview of the requirements findings An efficient way for communication between UGC producers and producers (live editors) - unidirectional messaging platform Scope & purpose What is to be achieved with this service? What is needed? The UGC producer should be able to receive information/hints about quality improvements of the recording (from the live editor). The live editor needs a communication channel to UGC producers to point out POI, which are not well covered yet. Inputs List of the required and optional input for the service Information about the UGC quality and hints for improvement Information about POI which are not well covered yet Outputs List of the output of the service Improved quality of UGC Better spatial coverage of event Flow Description of the event flow when the service is activated 1. Live editor sends hint for quality improvement message to UGC producer 2. UGC producer receives message containing hint 1. Live editor sends location with sparse coverage 2. UGC producer receives message containing location information Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)TBD Required resources Description of (external) resources involved POI and coverage information of the entire event Qualitative evaluation of UGC Dependencies



Software, hardware and platform dependencies Messaging platform inbuilt in smartphone application for UGC producers Messaging platform for live editors Related information Documents, links to the service

7.1.10 Social media integration

Contributing partners VRT Applies to use cases 5.2b Type Metadata

Summary Brief overview of the requirements findings Integrate social media into the project by making the social flow available to the application and setting up a service to communicate with different kinds of social media, using one aggregated REST interface. Social media information can be read, added, updated and removed. Scope & purpose What is to be achieved with this service? What is needed? Making social media data available for various services (e.g. Wall) via one aggregated and protected REST interface. Inputs List of the required and optional input for the service User authentication, administrative metadata (user information), social media data Outputs List of the output of the service Social media data Flow Description of the event flow when the service is activated - User authenticates himself using a public key or username and password (link do administrative metadata) - On successful authentication, the user is granted access to the social media REST service - User can connect with various social media (e.g. Facebook, Twitter) - User can read, add, update and remove social media data via REST (e.g. post, like, tweet) - Production can use interface get top trending tweeds, most likes etc.Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST using JSON definition language Required resources Description of (external) resources involved Administrative metadata service, social media resources (e.g. Facebook, Twitter) Dependencies Software, hardware and platform dependencies Database (e.g. MySQL), webserver (e.g. Apache), PHP or third-party (Firebase) Related information Documents, links to the service http://www.mysql.com http://php.net http://www.apache.org https://www.firebase.com

7.2 Capture Services

7.2.1 Calibration of technical equipment

Contributing partners BBC, iMinds Applies to use cases 1.6 Type Calibration



Summary Brief overview of the requirements findings Capture devices, in particular cameras and microphones, will need calibration before use to ensure the correct quality of the signals. As the output from multiple captures devices (e.g. individual cameras in a 360 degree rig) are to be combined, then calibration is required to ensure matched outputs and correct location information. Scope & purpose What is to be achieved with this service? What is needed? To ensure all the capture devices are operating correctly before the shooting starts, and their parameters have been measured. Inputs List of the required and optional input for the service Audio signals Video signals Location measurements Outputs List of the output of the service Position metadata for devices Capture device parameter metadata Correctly calibrated capture devices Flow Description of the event flow when the service is activated 1. Location scout records 3D locations of all the devices 2. Camera expert calibrates each camera, recording parameters 3. Audio expert checks microphone operation and helps set levels 4. Parameters for each are converted to capture metadata Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)The service shall support at least one of the following interfaces: * RESTful interface using JSON * ??? Required resources Description of (external) resources involved * Sound engineer for setting audio levels Dependencies Software, hardware and platform dependencies * Video & audio signal routing to check signals are reaching capture devices * Camera calibration software Related information Documents, links to the service

7.2.2 Capturing video

Contributing partners BBC, iMinds Applies to use cases 3.6, 2.3 Type Capture

Summary Brief overview of the requirements findings Various capture devices will generate video signals (professional video cameras, phones/tablets). These need to be captured into the ICoSOLE system. The video is required for video scene creation, live editing and post-production. Metadata generated by the capture devices should also be recorded with the video signals. Scope & purpose What is to be achieved with this service? What is needed? Video signals must be converted from the format generated by capture devices to a format understood by the ICoSOLE system. This applies both to professional and non-professional video capture devices.



For professional content capture using conventional equipment, an adaptor for HD-SDI capture hardware is required. Advanced/novel equipment may require an alternative adaptor interface, which could involve file-based or network stream-based inputs. Non-professional capture devices (UGC and semi-professional e.g. mobile devices or DSLR cameras) are likely to provide a file-based or network stream-based input. A network streaming and file upload interface is also required as part of this service. Digital media formats may require transcoding during capture, in order to be handled in the ICoSOLE system. Capture of professional and user-generated content will be differentiated. There will be significant differences in the type and set-up of recording devices. Professional equipment will be controlled by professional and trained users. User-generated recordings will be taken from mobile devices (smart phones, tablets etc.) with potentially untrained operators. Inputs List of the required and optional input for the service - Professional capture: number of cameras, their characteristics and locations - UGC capture: registered users, type of mobile recording devices, locations, timing information and other characteristics. - Digital video signals generated by the devices in various formats. Outputs List of the output of the service - Real-time streams of captured video signals in the ICoSOLE system. - Storage of captured video signals. - All meta-data directly derived from the recording device, associated with those streams/files. Flow Description of the event flow when the service is activated 1. Operator initiates video capture service on system 2. Service presents available video capture devices/hardware interfaces 3. Operator selects to add a new capture device 4. Operator specifies video hardware interface channels or stream URI for capture device 5. Service checks source and if it can be interpreted, creates new capture source 6. Service identifies available meta-data from device and attaches to source 7. Service begins local storage of video capture device stream 8. Service publishes video capture device stream availability to System Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema) REST using JSON representation. RTP and DASH streaming interfaces to other system components. Shared memory with other local processing services. Required resources Description of (external) resources involved Operator(s): Professional camera operator or UGC/semi-pro device operator Equipment: professional video camera(s), mobile recording devices, WAN/LAN for mobile streaming, global synchronisation source and distribution. Dependencies Software, hardware and platform dependencies All recording devices must be connected to the production platform. Special attention has to be paid to the different latencies, which might result from different connections (wired or wireless) to the platform. For professional video capture, the ICoSOLE system may need to support SDI I/O hardware. Related information Documents, links to the service For UGC capture, the operation of this service may be (partially) automated by the following services. R1.8 - Reading and storing UGC capture intentions R1.9 - Reading and storing UGC metadata This service is related to service R2.1, especially regarding UGC capture.



7.2.3 Capturing audio

Contributing partners BBC, DTO Applies to use cases 2.3a, 2.3b, 3.5, 3.6 Type Capture

Summary Brief overview of the requirements findings Various capture devices will generate audio signals (microphones, electronic instruments, phones/tablets, video cameras). These need to be captured into the ICoSOLE system. The audio needs to be available for use in scene creation, live editing and post-production. Metadata generated by the capture devices should also be recorded with the audio signals. Scope & purpose What is to be achieved with this service? What is needed? Audio signals must be converted from the format generated by capture devices to a format understood by ICoSOLE e.g. a stream of PCM digital audio sample data. This requires adaptors for analogue and digital audio formats common in professional broadcast systems. Other (UGC and semi-professional) devices may already generate digital audio signals in either streaming or file formats. A network streaming and file upload interface is also required as part of this service. Digital media formats may require transcoding during capture, in order to be handled in the ICoSOLE system. Capture of professional and user-generated content will be differentiated. There will be significant differences in the type and set-up of recording devices. It can be assumed that professional recording equipment will mainly be used in stationary conditions and controlled by professional, trained users, whereas user generated recordings will be taken from mobile devices (smart phones, tablets etc.) with untrained equipment holders. Need for professional capture: Studio-style microphone equipment, e.g. stereo, soundfield, Eigenmike, special mircrophone arrays. Need for consumer capture: Mobile recording devices, location information, and online connection to production platform. Inputs List of the required and optional input for the service - Professional capture: no of audio capture devices (including microphones), their characteristics and locations, audio interface hardware. - UGC capture: no of registered users, type of their mobile recording devices, their locations, timing information of active recordings, etc. - Analogue and/or digital audio signals generated by the devices in various formats. Outputs List of the output of the service - Real-time streams of captured audio signals in the ICoSOLE system. - Storage of captured audio signals. - All meta-data directly derived from the recording device, associated with those streams/files. Flow Description of the event flow when the service is activated 1. Operator initiates audio capture service on system 2. Service presents available audio capture devices/interfaces 3. Operator selects to add a new capture device 4. Operator specifies audio hardware interface channels or stream URI for capture device 5. Service checks source and if it can be interpreted, creates new capture source 6. Service identifies available meta-data from device and attaches to source 7. Service begins local storage of audio capture device stream 8. Service publishes capture device stream availability to System Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema) REST using xml data representation (compliant to a potentially defined scheme) or JSON representation. RTP and DASH streaming interfaces to other system components. Shared memory with other local processing services.



Required resources Description of (external) resources involved Professional capture: The service requires the availability of professional sound engineers in order to guarantee optimal quality and sound capture. UGC capture: The service involves a number of (semi-professional) recording equipment users, who have registered to the event before. The amount of users depends on the outspread of the event as well as on the number of volunteers, who applied to the capture request. Dependencies Software, hardware and platform dependencies All recording devices must be connected to the production platform. Special attention has to be paid to the different latencies, which might result from different connections (wired or wireless) to the platform. For professional audio capture, the ICoSOLE system may need to support common audio I/O systems on supported operating systems (e.g. CoreAudio, ALSA, ASIO). Hardware audio I/O cards are required for computers which require physical connection interfaces. Related information Documents, links to the service For UGC capture, the operation of this service may be (partially) automated by the following services. R1.8 - Reading and storing UGC capture intentions R1.9 - Reading and storing UGC metadata This service is related to service R2.1, especially regarding UGC capture.

7.2.4 Upload service

Contributing partners BIT

Applies to use cases 2.3a, 2.3b Type Capture

Summary Brief overview of the requirements findings Captured content will be uploaded (on-the-fly) to a central storage and processing unit on site. TBD: upload content in short segments or persistent stream? Scope & purpose What is to be achieved with this service? What is needed? Both, audio and video content needs to be transmitted from the recording device (smartphone) to an on-site processing unit. The quality (e.g. bitrate, resolution) of the transmitted content (at least for live scenarios) needs to be adapted to the maximum channel capacity (between user device and processing unit). Inputs List of the required and optional input for the service UGC from mobile devices Outputs List of the output of the service Content at central storage Flow Description of the event flow when the service is activated 1. A/V content is split into segments 2. Segments/Stream with appropriate size (and therefore quality) are/is uploaded to processing unit Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)TBD Required resources Description of (external) resources involved Transmission channel Dependencies Software, hardware and platform dependencies Smartphone application capable of



- recoding A/V content - split content in segments Related information Documents, links to the service

7.3 Processing

7.3.1 Automatic alignment of pictures taken on site survey

Contributing partners iMinds, JRS Applies to use cases 1.6, 3.3 Type Processing

Summary Brief overview of the requirements findings Semi-automatically place pictures in the scene based on measurements taken at the site-survey or operator intervention, in order to use them as a reference for aligning content captured during the production. Scope & purpose What is to be achieved with this service? What is needed? The purpose of the service is to verify, complete or update the location and orientation of image data taken during site survey. There are high requirements in terms of precision of this alignment, as it shall be used for validating, refining or correcting the spatial metadata of incoming content matching the images. In addition, annotations can be provided which can later be transferred to content visually overlapping with the pictures. Inputs List of the required and optional input for the service - A set of pictures - scene setup - measurements and annotations Outputs List of the output of the service - Location and orientation for each of the images - annotation of salient objects visible in the images Flow Description of the event flow when the service is activated - Get single or set of pictures - extract available metadata - match against other images, and estimate a confidence score - provide data for visualising results to the user - update metadata and annotations Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST using xml data representation (compliant to a potentially defined scheme) or JSON representation. Required resources Description of (external) resources involved Operator Scene description Image data set Dependencies Software, hardware and platform dependencies - Embedded location metadata or external measurements Related information Documents, links to the service



7.3.2 Automatic filtering of streams

Contributing partners JRS

Applies to use cases 3.2, 3.5 Type Processing

Summary Brief overview of the requirements findings Filter streams automatically based on available metadata (captured, annotated or automatically extracted) Scope & purpose What is to be achieved with this service? What is needed? The purpose of this service is to reduce the set of streams from which users are selecting content. Filtering is done using all metadata available for the essence streams, independent of their origin. Initially, filtering will be based on pre-defined rules. In a later stage, feedback from the subsequent tools in the production chain will be used, i.e. whether proposed streams are actually used or discarded, which location, objects, topics, etc. are selected. Inputs List of the required and optional input for the service - Available metadata streams related to essence streams in the current scene - predefined selection criteria (e.g. minimum quality requirements) - implicit and explicit feedback from authoring and editing tools (e.g. content selected/discarded) Outputs List of the output of the service - A reduced set of streams to be provided to the users for selection Flow Description of the event flow when the service is activated - Get metadata streams for current scene - check metadata against filtering criteria - make filter decision for a time window Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST using xml data representation (compliant to a potentially defined scheme) or JSON representation. Required resources Description of (external) resources involved Metadata streams Dependencies Software, hardware and platform dependencies Related information Documents, links to the service

7.3.3 Broadcast graphics service

Contributing partners TAW Applies to use cases 3.2 Type Processing

Summary Brief overview of the requirements findings The live editing application provides the editors with a selection of pre-produced broadcast graphics packages. The graphics service provides simple storage of these assets. Note: This service is not necessary if still images are part of the overall scene representation, which yet has to be determined. Scope & purpose What is to be achieved with this service? What is needed?



Within the ICoSOLE platform, broadcast graphics are simple still images defining the style of standard broadcast graphics elements. These are, for example, lower third backgrounds, station logos or panels for info insertions. We deliberately do not define advanced graphics formats in this context as this is out of scope of the ICoSOLE project. Inputs List of the required and optional input for the service - Pre-rendered static graphics from preproduction along with predefined type ("lower third", "logo", etc., "other") - Query to return all available still images of a certain type (URIs are returned) Outputs List of the output of the service - URIs of still images Flow Description of the event flow when the service is activated - Service is started - Preproduction registers a lower third background panel - Service returns an URI for the previous registration - Live editing queries service to return the URI of a lower third background panel - Service returns the URI of the previously registered background panel Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST API using JSON for RCP and other mime-types for retrieved graphics. Image resources are identified by an URI, as such any graphics can be registered and later retrieved by URI Required resources Description of (external) resources involved Images of pre-production Dependencies Software, hardware and platform dependencies Related information Documents, links to the service

7.3.4 Timeline alignment of content

Contributing partners DTO, iMinds Applies to use cases 3.2, 3.4, 5.1, 5.2a, 5.2b Type

Summary Brief overview of the requirements findings Time alignment of content will be provided timestamp based automatically by the system. If time alignment has to be changed, Operator and timestamp manipulation will be needed. Scope & purpose What is to be achieved with this service? What is needed? Live editing provides the editors with a global timeline and a time-coherent view of all incoming streams and media assets. Live production uses ad-hoc editing of the schedule and any eventual graphics content. This requires the underlying system to be able to perform in real time. However, the editing system also allows scheduling media on the global timeline later than the current live time. Off-site teams might be preparing a pre-scheduled segment, e.g. with a recap of the last two hours, using the same editing solution. The time alignment of content will be ensured by using timestamps within the media streams and a common clock system for all components. The use of timestamps and the possibility to change the timestamps enables the time alignment of content as well as it’s scheduling.



Inputs List of the required and optional input for the service - A common clock system - Media stream with timestamps - Interface for timestamp manipulation Outputs List of the output of the service - Media stream with manipulated timestamps Flow Description of the event flow when the service is activated - Default: System ensures the time aligned rendering and playout using the common clock System and timestamps - If the Operator decides to change the time alignment of content, the following steps will be necessary: - Selection of media stream - Display / observe timeline alignment - Determines time offset to be applied - Apply time offset to stream - Time offset will be recognized by all services using the stream Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST using xml data representation (compliant to a potentially defined scheme) or JSON representation. RTP and DASH streaming interfaces to other system components. Shared memory with other local processing services. Required resources Description of (external) resources involved Operator Common system clock Dependencies Software, hardware and platform dependencies - System clock - media streams with (correct) timestamps Related information Documents, links to the service

7.3.5 Live editing service

Contributing partners TAW Applies to use cases 3.2 Type Processing

Summary Brief overview of the requirements findings Editors create one or more linear programs from the available input material of the ICoSOLE platform (see use case 3.2). This service defines a high-level API for editing a program output in both live and scheduled mode. Scope & purpose What is to be achieved with this service? What is needed? Live editing will be done primarily via the live production desktop application. This application provides the editor with a graphical user interface (GUI) to produce the live program’s content. The implementation of this application is composed of several services provided by the ICoSOLE platform. The live editing service is one of these services, and defines the model of a linearly produced live program output. It also defines the operations that can be applied while the program is on air. As such, it acts as a high-level service on top of the live playout service. The live editing service exposes APIs for the following tasks:



- Managing editing sessions; - Managing program output; - Registering input media to be used for ad-hoc editing operations. In case of stream- or file-based media, caching for ad-hoc switching is considered and is configurable; - Configuring delivery mechanisms of edited program output (e.g. SDI output in parallel to publishing a live stream via MPEG DASH). This includes configuring the Metadata delivery mechanism; - Querying, adding and modifying editing actions along the program output’s timeline. Ad-hoc editing and scheduled operations should are modelled via a single API. Editing operations are: - Overlaying broadcast graphics (including animation and effects); - Applying effects based on multiple input streams: - Transitions; - Picture-in-picture; - Simple switching; As an optional enhancement the service allows multiple users to edit concurrently on the same program instance. This would additionally require APIs for user authentication and permission management. Also, editing operations need to carried out in transactions, and conflicting concurrent edits must be handled in some way. This service is not responsible for: - Querying the ICoSOLE scene representation; - Metadata filtering; - Providing on-demand playback capabilities; - Handling upstream events from playback clients (e.g. user-triggered actions via interactive playback);

Inputs List of the required and optional input for the service - Video/audio streams for being used as input to live program. - Locally attached SDI devices for interfacing with professional I/O equipment. - Editing actions carried out by editor. - Configuration of program output and delivery mechanisms. - Broadcast graphics.

Outputs List of the output of the service - One or more program outputs delivered via one ore more delivery mechanisms (SDI, stream, file). - Where possible, delivery formats include metadata as well.

Flow Description of the event flow when the service is activated (This flow describes only a simple example) - The live editing service is started. - Service provides a list of active program outputs and currently running editing sessions. - No programs or editing session are running yet. - Client creates and configures a program output. - Client attaches and enters an editing session to the previously created program output. - Client registers multiple URIs of media for being used as live input or (which have been retrieved by the client via another service layer). - Service pre-rolls these URIs in order to guarantee low-latency operations on these input streams. - Client registers broadcast graphics for being used as overlays. - Service caches and pre-renders graphics for being used in the current configuration of the program output. - Client commits the following editing actions: - Switch program output to another already registered input stream. - At the same time, overlay the previously added broadcast graphics. - Service performs the previously committed editing actions. - Client registers the URIs of pre-produced segments for the next hour. - Client schedules the playback of the pre-produced segments for a particular time of the program



output. - Client exits and destroys the editing session. Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST using JSON representation. SDI devices for professional I/O equipment DASH / RTMP / RTP used for media delivery and input media (to be determined depending on latency and standards requirements e.g. for metadata) File-based I/O Required resources Description of (external) resources involved Previously recorded and processed live or offline media. Professional equipment. Definition of an even higher-level workflow such as the live production application. Dependencies Software, hardware and platform dependencies The implementation of this service needs to have a low-level playout service available (see live playout service). The capabilities of the playout service define which effects, input media formats and delivery formats can be exposed vie the live editing service. Related information Documents, links to the service

7.3.6 Editing Scene Representation

Contributing partners BBC Applies to use cases 3.3, 3.4 Type Processing

Summary Brief overview of the requirements findings Metadata will be created from a number of sources (automatically and manually generated) and associated with the various audio and video streams within the system. This metadata contains both information (such as capture information) and control data (such as desired audio rendering positions and parameters). Therefore the metadata may be incorrect or need correction or be edited as part of the creative process. Scope & purpose What is to be achieved with this service? What is needed? To be able to check, adjust and edit metadata and monitor the effects of such changes. Inputs List of the required and optional input for the service Audio streams Video streams Metadata streams Annotations associated with metadata Outputs List of the output of the service Metadata streams Audio and/or video streams for monitoring purposes Flow Description of the event flow when the service is activated 1. Operator selects metadata stream(s) 2. System optionally selects audio and/or video streams associated with metadata 3. Operator reviews/checks metadata (against content if possible) and makes adjustments if necessary4. Operator chooses to monitor audio and/or video associated with metadata 5. Operator edits control parameters of metadata (audio rendering controls for example) and monitors results of changes 6. Updated metadata streams are outputted to system Interface



Interface & definition language used to communicate with the service (e.g. REST using JSON Schema) The service shall support at least one of the following interfaces: * RESTful interface using JSON * WebSockets using a payload format TBD Required resources Description of (external) resources involved * Operator (Producer / Mixing Engineer) Dependencies Software, hardware and platform dependencies * Network communications supporting TCP (high bandwidth if video monitoring is required) * Display and optionally audio monitoring facilities * User interface * Access to the system architecture Related information Documents, links to the service

7.3.7 Automatic extraction of metadata

Contributing partners JRS, DTO Applies to use cases 3.3, 3.4 Type Processing/metadata

Summary Brief overview of the requirements findings Extract metadata about quality and description of content (e.g., detection and classification of content) from audio-visual data, and provide annotations. Scope & purpose What is to be achieved with this service? What is needed? The purpose is to extract information that can support filtering and selection of content. NOTE: This is a generic placeholder for specific metadata extraction services, that will differ in terms of input needed, latency and the type of metadata produced. Inputs List of the required and optional input for the service - One or more stream(s) of a/v essence - related metadata streams Outputs List of the output of the service - One or more metadata stream(s) with time-based annotations Flow Description of the event flow when the service is activated - Read a time window of a/v content - output results for a time window Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST using xml data representation (compliant to a potentially defined scheme) or JSON representation. Required resources Description of (external) resources involved - Essence and metadata streams - configurations of AME tools Dependencies Software, hardware and platform dependencies - Embedded location metadata or external measurements Related information



Documents, links to the service

7.3.8 Semi-automatic linking of content with metadata

Contributing partners DTO Applies to use cases 3.2, 3.3, 3.4, 5.2b Type

Summary Brief overview of the requirements findings Metadata can be used to describe either semantic or non-semantic information, e.g. technical information. To enrich the AV content, metadata within a separate stream are an appropriate (transport) medium, carrying dynamic as well as static metadata. Therefore, editing and handling with metadata as well as content-based metadata creation and linking are needed. Scope & purpose What is to be achieved with this service? What is needed? For live editing, metadata information that is either carried with the media data themselves or made available by metadata management is used for filtering a selection of streams automatically. The content of additional simple graphics instantaneously added to the live content can depend on metadata, or other data streams to which the live editor has access. Metadata streams carried with media data, for instance created by scene creation process, are passed to the user endpoint along with the audio and video. The handling of metadata before and during the production requires its integration in the scene representation. Besides the monitoring of storing content and metadata, the preparation of content for selection and scene creation by putting into context, updating/correcting metadata and linking with other material is beneficial. This process can be supported by automatic metadata extraction from the incoming content, and automatic context insertion by semi-automatic linking with production data. Meta-data required for the semantic scene representation or all the information that is needed to allow interactive experiences is also created and included in the scene model. The metadata captured with the content, added by UGC contributors or extracted automatically will be manually reviewed and possibly amended/corrected. Due to the constraints of live production the latter will often only consist in flagging metadata as reliable/seemingly correct or not. In addition, locations or key persons may be annotated. Besides the semantic metadata processing, a specific focus in metadata editing will be checking and possibly adjusting the spatial composition of a/v streams in the scene. For nearby of overlapping streams, options for transitions can be set or modified. To represent and render an immersive audio scene, creation of specific metadata is required, which will configure the user set-up. Inputs List of the required and optional input for the service - AV-streams - Metadata streams or information - User Control information stream Outputs List of the output of the service - Filtered Metadata streams or information with tags to which content it belongs Flow Description of the event flow when the service is activated The general flow of semi-automatic linking of content is a very complex process and takes place in several different parts of the system. E.g. a semantic exploitation algorithm generates information, e.g. the detection of a person that will be used in a later processing step. Therefore, this requirement can be regarded as a concept. Nevertheless, some steps for monitoring or editing could be defined as general processing steps within the (live) editing process: Step 1: Selection of a (content) stream



Step 2: Applying an exploitation algorithm OR Checking / editing of metadata Step 3: Monitoring the results Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST using xml data representation (compliant to a potentially defined scheme) or JSON representation. RTP and DASH streaming interfaces to other system components. Shared memory with other local processing services. Required resources Description of (external) resources involved Semantic web queries for static information Event (Set) database information Dependencies Software, hardware and platform dependencies Related information Documents, links to the service

7.3.9 Audio processing

Contributing partners BBC, DTO


Summary Brief overview of the requirements findings It is necessary to provide the ability to analyse, process and mix audio as part of the creative process. In contrast to current broadcast mixing systems, ICoSOLE shall allow the use of metadata (both created internally and externally) to drive and control such processes. The use of analysis as feed-forward and feedback control shall be considered particularly important. Scope & purpose What is to be achieved with this service? What is needed? The service shall include provision for EQ and DRC processing. Inputs List of the required and optional input for the service A single audio stream A single metadata stream Outputs List of the output of the service A single audio stream Flow Description of the event flow when the service is activated The service shall receive a single audio stream, a single metadata stream containing control data and produce a single audio stream of processed audio. Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)There shall be NO interface to this service, all control is via the metadata stream Description of (external) resources involved None Dependencies Software, hardware and platform dependencies * Network communications supporting TCP * Network communications supporting UDP * High performance audio processing



Related information Documents, links to the service

7.3.10 Audio analysis

Contributing partners BBC, DTO


Summary Brief overview of the requirements findings It is necessary to provide the ability to analyse, process and mix audio as part of the creative process. In contrast to current broadcast mixing systems, ICoSOLE shall allow the use of metadata (both created internally and externally) to drive and control such processes. The use of analysis as feed-forward and feedback control shall be considered particularly important. Scope & purpose What is to be achieved with this service? What is needed? The analyses required shall include loudness, Mel Frequency Cepstral Coefficient (MFCC) calculation, and spectral and dynamic range. Support for VAMP plugins shall also be supported. Inputs List of the required and optional input for the service A single audio stream Outputs List of the output of the service A single metadata stream Flow Description of the event flow when the service is activated The service shall receive a single audio stream, pass it to the desired audio analysis tool and generate a metadata stream with the results of the analysis contain within it. Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)The service shall allow selection and control of parameters specific to each audio analysis tool via a RESTful service and/or a WebSocket Description of (external) resources involved * Editor / producer / mixing engineer Dependencies Software, hardware and platform dependencies * Network communications supporting TCP * Network communications supporting UDP * High performance audio processing * Graphical User Interface Related information Documents, links to the service

7.3.11 Audio mixing

Applies to use cases 3.2, 3.4 Type

Summary Brief overview of the requirements findings It is necessary to provide the ability to analyse, process and mix audio as part of the creative process. In contrast to current broadcast mixing systems, ICoSOLE shall allow the use of metadata (both created internally and externally) to drive and control such processes. The use of analysis as feed-forward and feedback control shall be considered particularly important. Scope & purpose What is to be achieved with this service? What is needed?



Audio mixing shall allow multiple audio sources to be combined at different levels and panning positions. However, processing may affect the metadata only with no actual change to the audio, with the information about the processing to be applied added to the metadata for downstream processing. This allows processing to occur at the most appropriate stage in the stream handling, even if that is the user's device. Inputs List of the required and optional input for the service Multiple audio streams Multiple metadata streams Outputs List of the output of the service A single audio stream A single metadata stream Flow Description of the event flow when the service is activated The service shall receive multiple audio streams, multiple metadata streams which describe the levels and panning positions of the audio streams and optionally produce an audio stream and/or a metadata stream containing the mixing information. Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)The service shall allow basic control of levels and panning positions via a RESTful service and/or WebSocket Description of (external) resources involved * Editor / producer / mixing engineer Dependencies Software, hardware and platform dependencies * Network communications supporting TCP * Network communications supporting UDP * High performance audio processing * Graphical User Interface Related information Documents, links to the service

7.3.12 Headphone sound rendering

Contributing partners BBC Applies to use cases 3.2, 3.4, 5.1, 5.2a-c Type Processing

Summary Brief overview of the requirements findings Binaural processing techniques will be applied to the rendering of immersive headphone audio. This is required both in production and audience-facing applications, each with differing interfaces. Audience-facing rendering will be performed on a mobile platform, either as a native application or using the web browser. Production rendering will be carried out on a workstation computer. Incoming audio signals (in object-, channel- and/or scene-based formats, with format and control metadata) will be rendered to a two-channel signal for headphones in real-time. Rendering will be capable of dynamic adjustment to listener orientation data (i.e. from a head tracker). The rendering must be to a high quality so as not to introduce artefacts that degrade the quality of the listening experience. Personalisation parameters (e.g. controlling impulse response data or equalisation) are also incorporated into the rendering to enhance the listening experience. Scope & purpose What is to be achieved with this service? What is needed? Real-time dynamic binaural rendering of 3D audio streams (essence plus meta-data) to headphones, for production and audience listening. Inputs



List of the required and optional input for the service 3D audio stream (audio essence plus meta-data) Auralisation data (impulse responses and environment parameters) Listener position/orientation data Personalisation parameters Outputs List of the output of the service Headphone audio signal Flow Description of the event flow when the service is activated 1. Audio stream delivered to binaural renderer 2. User connects to service 3. User connects headphones and tracker 4. Service loads saved renderer settings 5. User starts playback/rendering 6. User adjusts renderer settings if needed 7. User enjoys the binaural sound Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST interface using JSON schema (for control of server-side parameters) RTP and MPEG DASH delivery of object-based audio Web browser GUI for user control AES X212 files for impulse responses Local interface to listener transformation data (tracking data) Required resources Description of (external) resources involved DASH delivery of content, playout platform for users, advanced audio scene authoring Dependencies Software, hardware and platform dependencies Smartphone application, headphones and head tracking device, PC application, audio DAC, network communication supporting TCP and UDP, high performance audio processing. Related information Documents, links to the service

7.3.13 Loudspeaker sound rendering

Contributing partners BBC Applies to use cases 3.2, 3.4, 4.1, 4.2 Type Processing

Summary Brief overview of the requirements findings Audio signals (including 3D object-based audio scenes) will be rendered to loudspeakers for purposes of production monitoring and for audience listening on TV platforms. Production rendering will be carried out on a workstation computer whereas audience-facing rendering will be carried out in the web browser. Incoming audio signals (in object-, channel- and/or scene-based formats, with format and control metadata) will be rendered to loudspeakers in real-time. A range of loudspeaker layouts will be supported, with the ability to specify custom loudspeaker positions. Personalisation parameters (e.g. controlling impulse response data or equalisation) are also incorporated into the rendering to enhance the listening experience. Scope & purpose What is to be achieved with this service? What is needed? Real-time rendering of 3D audio streams (essence plus meta-data) to any defined loudspeaker setup, for production and audience listening. Inputs List of the required and optional input for the service



3D audio stream (audio essence plus meta-data) Loudspeaker setup data Personalisation parameters Outputs List of the output of the service Loudspeaker audio signals Flow Description of the event flow when the service is activated 1. Audio stream delivered to loudspeaker renderer 2. User connects to service 3. Service loads saved renderer settings including loudspeaker layout(s) 4. User adjusts renderer settings if needed 5. User starts playback/rendering 6. User enjoys the loudspeaker sound Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST interface using JSON schema (for control of server-side parameters) RTP and MPEG DASH delivery of object-based audio Web browser GUI for user control Required resources Description of (external) resources involved DASH delivery of content, playout platform for users, advanced audio scene authoring Dependencies Software, hardware and platform dependencies TV platform application, loudspeakers and amplification, audio DAC, PC application. Related information Documents, links to the service

7.3.14 Media coding

Contributing partners BBC Applies to use cases 2.3a, 2.3b, 3.2, 3.5, 4.2, 5.1, 5.2a, 5.2b, 5.2c Type Processing

Summary Brief overview of the requirements findings Various contributing media devices, services and user endpoints will present and require audio and video in different formats. Therefore a service is required to convert between the different formats, 'transcoding' the media. Scope & purpose What is to be achieved with this service? What is needed? The service shall receive video and/or audio in one format, combine, split or transcode into another format. This service shall be employed whenever a service requires media from another service but that service does not provide the media in the correct format Inputs List of the required and optional input for the service One of: 1. A single multiplexed media stream containing at most one video stream and multiple audio streams 2. Non-multiplexed media streams with at most one video stream and multiple audio streams Outputs List of the output of the service One of: 1. A single multiplexed media stream containing at most one video stream and multiple audio streams 2. Non-multiplexed media streams with at most one video stream and multiple audio streams Flow Description of the event flow when the service is activated



The service receives the input(s) and processes the video streams and/or audio streams, transcoding and (de-) multiplexing where necessary to produce the desired output stream(s) Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)The description of what output(s) are required shall be passed to the service via a RESTful service Description of (external) resources involved Dependencies Software, hardware and platform dependencies * Network communications supporting TCP/UDP * High performance video/audio processing Related information Documents, links to the service

7.3.15 Audio scene authoring interface

Contributing partners BBC Applies to use cases 3.4 Type Processing

Summary Brief overview of the requirements findings In order to generate immersive audio scenes that can be reproduced on many platforms, object-based representations of audio scenes will be used, representing content as audio essence plus (control and descriptive) meta-data. A service is required to allow a sound engineer to author these object-based audio streams to create immersive audio experiences. Captured and processed audio streams will be arranged in a scene using a graphical user interface and physical control interface. This will generate the required (control) meta-data and produce an object-based audio stream. This service will use audio rendering services to allow the sound engineer to monitor the result of their work in real-time. Scope & purpose What is to be achieved with this service? What is needed? Authoring of 3D object-based audio scenes using a graphical user interface and physical control interfaces. Inputs List of the required and optional input for the service Audio source signals Descriptive meta-data for sources and set (sub-event) Outputs List of the output of the service Object-based 3D audio scene stream (audio essence plus meta-data) Flow Description of the event flow when the service is activated 1. Sound engineer connects to service 2. Sound engineer selects relevant audio source signals from those available for inclusion in scene 3. Using GUI and physical controls, sound engineer modifies control meta-data including source position and interactivity parameters (such as foreground/background classification) 4. Object-based 3D audio scene stream is generated Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema) Required resources Description of (external) resources involved Audio processing, analysis and mixing tools. Binaural and loudspeaker rendering. Event calendar, 3D event model, content metadata extraction and linking, editing scene representation, timeline alignment of content.



Dependencies Software, hardware and platform dependencies Physical controllers, audio monitoring equipment, workstation computer with display Related information Documents, links to the service

7.3.16 Video scene authoring interface

Contributing partners JRS Applies to use cases 3.7 Type Processing/metadata

Summary Brief overview of the requirements findings Provide a user interface for authoring the video scene Scope & purpose What is to be achieved with this service? What is needed? Allow user to make adjustments to the video scene, select streams (based on the output of automatic filtering), defined transitions and assign video streams to specific distribution channels. The UI has to be capable to support the user under the constraints of the live production, and thus needs to provide few options that can be easily selected. Inputs List of the required and optional input for the service - The (filtered) representation of the captured scenes Outputs List of the output of the service - A subset of selected streams - metadata on transitions - metadata describing intended use of streams for different distribution channels Flow Description of the event flow when the service is activated - Visualise options to the user - continue to output based on automatic settings and previous decisions until user interventions are received - update metadata based on user interactions Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)Web-based user interface Required resources Description of (external) resources involved - Scene representation Dependencies Software, hardware and platform dependencies Related information Documents, links to the service

7.4 Distribution services

7.4.1 Synchronisation signal distribution

Contributing partners BBC Applies to use cases 3.2, 3.3, 3.4, 3.6, 3.7 Type Distribution

Summary



Brief overview of the requirements findings In order to ensure that all media streams have the same sense of time and so can be synchronized at any point in the system, a mechanism to provide reliable, accurate and precise timing to parts of the system is required. Scope & purpose What is to be achieved with this service? What is needed? The service shall provide ns accurate timing synchronized to a central point (possibly through layers of synchronization systems). This service shall be used to 'timestamp' packets of media that form media streams, metadata that form metadata streams and control value changes that may affect other systems Inputs List of the required and optional input for the service A synchronization and timing source Outputs List of the output of the service Timing data applied to streams and control value changes Flow Description of the event flow when the service is activated The service receives the input(s) and processes the video streams and/or audio streams, transcoding and (de-) multiplexing where necessary to produce the desired output stream(s) Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)There shall be no interface provided to this system for control. Configuration shall be via a RESTful service but shall be assumed to be rarely changing Description of (external) resources involved Dependencies Software, hardware and platform dependencies * Network communications supporting TCP/UDP Related information Documents, links to the service

7.4.2 Signal routing in production system

Contributing partners BBC Applies to use cases 2.3a, 2.3b, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7 Type Processing

Summary Brief overview of the requirements findings The ICoSOLE production system will be a distributed set of computing nodes that can perform different processing tasks. A service is required to configure the routing of signals (media and meta-data streams) between these nodes to allow a production chain to be realised. This service should be aware of the available nodes in the system and the media streams that are available to be routed. Scope & purpose What is to be achieved with this service? What is needed? Signal routing service for the ICoSOLE production system, to allow control of distribution of signals between processing nodes. Inputs List of the required and optional input for the service System nodes publishing available streams Outputs List of the output of the service Control of signal streaming between nodes. Flow



Description of the event flow when the service is activated 1. Service presents available nodes and streams, as well as existing routing 2. User selects to route a stream from one node to another 3. Service configures nodes to make the connection and updates its routing display 4. Any new system nodes publish their availability to the service 5. The display of available nodes is updated Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)RESTful interface between service and system nodes. Network streaming over TCP and/or UDP (e.g. RTP or DASH) Scene model for nodes and signal streams. Required resources Description of (external) resources involved Media coding Dependencies Software, hardware and platform dependencies Fast IP network Related information Documents, links to the service

7.4.3 Production data storage

Contributing partners BBC Applies to use cases 2.3, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7 Type Processing

Summary Brief overview of the requirements findings A service for storage of ICoSOLE data in the production system is required to allow processing and editing of broadcast content from captured data both in live and offline scenarios. This includes short-term local storage and longer-term centralised storage. The stored media should then be accessible to other production services. Scope & purpose What is to be achieved with this service? What is needed? Storage of generated media and meta-data in the ICoSOLE production system. Inputs List of the required and optional input for the service Media and metadata streams Outputs List of the output of the service Access to stored data Flow Description of the event flow when the service is activated 1. Generated media and data streams on a processing node in the production system are stored locally with all related meta-data including timing information, whilst services are running. 2. This local store of data can be accessed by local services if required 3. Error tolerant transfer to a central storage node is run in the background. 4. Central store of data can be accessed by all services Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)RESTful interface for control and publishing list of stored content DASH streaming of captured content between local and central storage Required resources Description of (external) resources involved



Dependencies Software, hardware and platform dependencies Fast IP network Related information Documents, links to the service

7.4.4 Live playout service

Contributing partners TAW Applies to use cases 3.2 Type Playout

Summary Brief overview of the requirements findings This service provides a low-level interface to a media-processing engine that is able to output a live program based on the linear composition of input media and effects. This service is intended to be mainly used by the implementation of the live editing service. As such, this service provides a media-processing graph in which audio and video data can be read, processed and written synchronized to a common timeline. Scope & purpose What is to be achieved with this service? What is needed? This service provides an API around the model of a media-processing graph that is able to operate both in real-time and offline modes. This graph provides sources and sinks for many different formats (e.g. mpeg-ts) carried over different transport mechanisms (e.g. http server in case of MPEG-DASH), and also provides I/O capabilities of devices (e.g. SDI devices). Additionally, a pre-defined collection of standard codecs to transcode between compressed and uncompressed formats is available. Effects are exposed via processing nodes, which are placed in-between sources and sink nodes. The effects' parameters can be hooked onto a global timeline. Where possible, hardware acceleration for effects (via OpenGL) and en/decoding is available. The implementation relies on the open-source media framework GStreamer; hence the service's capabilities can be extended by writing custom plugins or modifying existing ones, when necessary. The primary application of this service is to provide the low-level media operations that are necessary to produce and edit a live TV program (see live editing service). However, due to the generic nature of the media-processing graph, it could be applied to other use-cases as well (e.g. ingest, transcoding, metadata extraction, and so forth). The following items are out of the scope of this service: - Persistent data storage of pipeline configurations. - Asset management, i.e. the playout service does not keep track of any repositories. Inputs List of the required and optional input for the service - Configuration of a media graph and its node properties (source nodes -> processing nodes -> sink nodes). - URIs of input assets (audio, video, graphics) - Master clock that drives the graph's timeline - Time-driven modifications of node properties or manipulation of the graph structure itself. Outputs List of the output of the service - Output streams as being sent to sink nodes (SDI, file, stream-based…), i.e. the output of the media graph. - Feedback on the compatibility of connecting two nodes (format negotiation, state validation...) - Messages of events that occurred on the graph's timeline Flow Description of the event flow when the service is activated - Services start without any media graphs currently running. - Client creates a new graph - Client adds a source node and configures its input URI (e.g. pointing to a MPEG-DASH stream)



- Client adds a sink node (e.g. RTMP publish point of web streaming server) - Client adds a processing node to overlay graphics on top of the video stream - Client connects the source with processing, the processing with the sink node. - Client starts the graph - Service initializes the nodes, validates the connections and transitions the graph into "playing" state. - Service sends a message to the client that the graph has successfully transitioned to "playing" state. - Service buffers the source node's input stream data. - Service lets the sink node (configured as running in real-time mode) pull data in real time from its predecessors. - Service renders the media graph repeatedly for each requested video frame (and audio packet). - Client adds a new source node, now using an SDI input device. - Client reconfigures the graph to use the newly created SDI source node instead of the MPEG-DASH source node. - Service carries out the requested re-configuration of the graph. - Service sends a message to the client that the graph has successfully switched the input source. - ... Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)Low-latency inter-process communication protocol (to be determined, current candidate is zeroMQ + compressed JSON). REST does not fulfil the requirements of asynchronous communication and low-latency communications that is necessary to properly interact with this service. Using a messaging framework like zeroMQ, messages can be sent in-process, inter-process and over TCP/IP. Required resources Description of (external) resources involved Dependencies Software, hardware and platform dependencies GStreamer library. Possible use of third-party plugins for GStreamer (to be determined) zeroMQ (to be determined), C++ JSON parsing library (to be determined), eventually messagepack for compression Hardware platform with OpenGL support and optimally H.264 hardware encoding support. Related information Documents, links to the service http://zeromq.org http://gstreamer.freedesktop.org http://msgpack.org

7.4.5 Scalable IP-based distribution scheme for live playback

Contributing partners iMinds Applies to use cases 3.2, 4.1, 5.1 Type Distribution

Summary Brief overview of the requirements findings Live playout of (professional) broadcast content mandates the simultaneous delivery of identical (high-quality) material to large numbers of viewers. As the ICoSOLE system also targets IP-based media distribution, the demand for an efficient, IP-friendly content distribution method arises. Due to the poor scalability properties of unicast traffic, the content dissemination scheme will need to technologically surpass one-to-one connections. Also, in order to improve the delivery efficiency, the service will adopt efficient caching schemes and CDN-like functionality which do not interfere with the chosen streaming method and which are located close to the end-user in the network topology. The service will support heterogeneous types of video content (e.g., traditional video, omni-directional or panoramic) and the distribution of the elementary streams will preferably be MPEG-DASH-compliant. Scope & purpose What is to be achieved with this service? What is needed? The service aims to offer an efficient IP multicast-based distribution engine for live content that allows the dissemination of professional "broadcast" footage to a large amount of subscribers simultaneously, in such a way that the load on the underlying IP infrastructure is minimized. The ultimate objective here is to design a scalable solution that is able to optimize the ratio of concurrent client count versus



network resource consumption, preferably in the context of MPEG-DASH-based content delivery. In case a consumer's access connection would not be IP multicast-enabled, the involved client will be serviced by falling back to a unicast-only scheme that is enhanced with CDN-like caching functionality. Finally, the service will include functionality to enable fast "zapping" between broadcast streams. Inputs List of the required and optional input for the service The service requires access to the results of the audio-visual capture of the real-world event. This envelops access to the output of the live editing task pertaining to the real-world event (i.e., the live broadcast signals that are generated by the ICoSOLE content producer(s)). Furthermore, metadata about the event and its involved elementary streams might also be required as input, as such information is likely to offer opportunities to optimize the delivery of the audio-visual footage. Outputs List of the output of the service The elementary streams are in real time delivered to a consumer base that is quantitatively as large as possible, this way effectively emulating broadcast-based content distribution. Flow Description of the event flow when the service is activated 1. The consumer connects to the ICoSOLE system on his TV set at home to consume the professional capture of the spatially outspread live event that is broadcasted in real time. 2. A consumer tunes in to a live ICoSOLE "broadcast" stream. 3. The service ascertains whether or not it is able to disseminate the involved stream to the involved client using IP multicast. 3.1 If not (e.g., the consumer's ISP lacks last-mile multicast support), the service delivers the live broadcast signal to the consumer using IP unicast. CDN-like caching functionality that is deployed topologically close to the consumer is exploited in order to improve the unicast delivery. 3.2 If so, the appropriate actions are undertaken to start delivering the broadcast content to the new client using IP multicast. This might involve, for example, adding the client to the multicast group that is responsible for the distribution of the broadcast stream at hand. 3.2.1 In order to overcome large start-up delays due to slow BGMP (Border Gateway Multicast Protocol) message propagation along the end-to-end network path that connects the consumer to the content distribution server, the service might resort to IP unicast to deliver the initial fragments of the broadcast stream to the client. As soon as the multicast path is set up and ready to use, the service would then start serving the client using IP multicast. 4. Step 3 is repeated each time the consumer "zaps" to another broadcast stream. 5. The consumer decides to abandon the live ICoSOLE broadcast. The service reacts appropriately (e.g., by freeing now unexploited network resources). Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)No externally accessible interface is envisioned for this service. Required resources Description of (external) resources involved * The IP-based network infrastructure that will implement the content distribution. * The live editing platform that is responsible for producing the live broadcast signals. * Optionally: Metadata about the event and its involved elementary streams, as such resources might offer opportunities to optimize the delivery of the audio-visual footage. * If necessary, a live broadcast environment will be simulated by resorting to static, file-stored footage that is streamed in a fashion similar to live content. This need will arise when interfacing with the live editing component turns out to be problematic (e.g., during the development phase when both the live editing tool and the scalable distribution engine are still actively being developed). Dependencies Software, hardware and platform dependencies * As the content dissemination will be IP-based, this service depends on the presence of IP networking infrastructure. * The delivery of the elementary streams will preferably be MPEG-DASH-based in order to maintain uniformity and consistency with the other playout-related ICoSOLE services. However, in case the deployment platform (i.e., the TV set) would lack MPEG-DASH support, a fallback will need to be made to a streaming technology that is actually supported by the involved platform and which ideally exhibits a feature set similar to MPEG-DASH.



Related information Documents, links to the service

7.4.6 Adaptive bit-rate media delivery

Contributing partners BIT, iMinds Applies to use cases 4.1, 4.2, 5.1, 5.2a, 5.2b Type Distribution

Summary Brief overview of the requirements findings Multimedia content has to be broadcasted to consumers in an efficient way. The quality of audio-visual content (e.g. resolution, bitrate...) should adapt to the current network capacity of the consumer. Scope & purpose What is to be achieved with this service? What is needed? A DASH based media delivery for audio/video/(subtitles?) content is needed to maximize QoE for consumers. Inputs List of the required and optional input for the service Audio and video segments stored in various representations (server-side) Outputs List of the output of the service Audio and video segments in the highest representation the network is capable to transmit (client-side) Flow Description of the event flow when the service is activated 1. DASH client application requests segments for dedicated time 2. While downloading segment, channel capacity is estimated 3. Choose appropriate representation for next segments according to 2. Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema) Required resources Description of (external) resources involved Transmission channel Dependencies Software, hardware and platform dependencies • A scalable IP-based distribution scheme that is able to deliver multimedia content to a large quantity of subscribers simultaneously • DASH client application • DASH transcoding server application Related information Documents, links to the service

7.5 Audience services

7.5.1 Playout platform for mobile users

Contributing partners BIT, iMinds Applies to use cases 5.1, 5.2 Type Audience

Summary Brief overview of the requirements findings The event can be experienced live or on-demand with the ability to step back and forth (both on the timeline and spatially) and reviewing multiple clips of dedicated scenes from different angles. A summary (composed by the production crew) can be watched or one can browse and select all available content (sorted by time, location and performing artist). Controlling audio functionalities like crowd noise and commentary adjustment and choosing an alternative position in the audience with



binaural audio experience is provided. Scope & purpose What is to be achieved with this service? What is needed? A smartphone application with an intuitive GUI is needed to make multimedia content (live and on-demand) available for consumers. An efficient way to browse the content (both spatially and on the timeline) is needed. Dynamic binaural audio streams are needed with the use of head tracking, affording an immersive headphone sound experience. Inputs List of the required and optional input for the service The network streams representing the multimedia content User input in order to control the media presentation (e.g., content/stage selection, view and perspective selection, navigation between spatially dispersed yet temporally parallel sub-events) Head tracking device input for binaural rendering Metadata about the event

Outputs List of the output of the service Satisfied consumer Flow Description of the event flow when the service is activated 1. The consumer accesses the playout platform on his/her mobile device 2. Multimedia content is delivered to the consumer using DASH 3. The consumer interacts with the platform via a simple, intuitive Graphical User Interface (GUI) a. to switch between different locations/stages b. to step back and forth on the timeline c. to select/lock to POI d. to watch a summary (composed by the production crew) 4. The consumer ends his/her session Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)TBD Required resources Description of (external) resources involved - Dependencies Software, hardware and platform dependencies Smartphone application with GUI Audio equipment for binaural audio experience Related information Documents, links to the service

7.5.2 Interactive Wall

Contributing partners VRT Applies to use cases 5.2b Type Metadata

Summary Brief overview of the requirements findings This service provides access to the Wall backend data via a common (REST) interface. User actions in the Wall frontend are translated (e.g. using JavaScript) to REST calls and processed by this service. Also, video streams of different qualities and resolutions are delivered to the Wall via DASH by this service. Scope & purpose What is to be achieved with this service? What is needed? Making Wall backend data available and editable for frontend applications. This can be metadata (e.g. user information, social media, location) and data (e.g. video and audio streams).



Inputs List of the required and optional input for the service User authentication, interactive Wall metadata Outputs List of the output of the service Interactive Wall metadata and data Flow Description of the event flow when the service is activated - User authenticates himself using a username and password (link do administrative metadata) or public key (production crew) - On successful authentication, the user is granted access to the Wall backend REST service - User can list, read, add and remove Wall metadata via REST - User can subscribe to Wall events using a JavaScript library - User can request A/V streams in different qualities and resolution (DASH delivery) Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST using JSON definition language Required resources Description of (external) resources involved Administrative metadata service, UGC metadata service, message service, calendar metadata service, social media service, DASH delivery of content (semantic scene model) Dependencies Software, hardware and platform dependencies Database (e.g. MySQL), webserver (e.g. Apache), PHP, JavaScript event library or third-party (Firebase) Related information Documents, links to the service http://www.mysql.com http://php.net http://www.apache.org http://jquery.com https://www.firebase.com

7.5.3 Playout client for live playback on TV

Contributing partners iMinds, BBC Applies to use cases 4.1 Type Playout

Summary Brief overview of the requirements findings Provide the possibility to explore, on a TV set in a home environment, the full range of available professional content in an interactive and immersive way, hereby also leveraging the provided metadata. The involved audio-visual content must be of a high quality level and its network distribution will preferably be MPEG-DASH-based. The service will support heterogeneous types of video content (e.g., traditional video, omni-directional or panoramic). The playout client includes stream pre-caching provisions to reduce the latency when switching between constituting elementary flows. Advanced video presentation features are also envisioned in order to enhance the immersive experience for the consumer at home; examples include side-by-side or picture-in-picture visualization schemes, and effective tools to interact with omni-directional or panoramic visual footage. Offering efficient and potentially immersive means that simulate physical traversal of the spatially outspread event is also targeted. Scope & purpose What is to be achieved with this service? What is needed? The service's objective is to give a home user, in real time, an impression of the spatially outspread live event that is as immersive as possible. The ultimate goal consists of emulating a sense of "being there" for home consumers (i.e., making the viewers feel like they are physically present at the covered event, while in reality they are not). Notice that this service solely targets professional "broadcast" content;



UGC is deliberately not considered. The service requires access to the output of the live editing task, as well as to relevant metadata pertaining to the real-world event. The service's video rendering features will surpass classic video by also considering, for example, omni-directional or panoramic material. Inputs List of the required and optional input for the service The service requires access to the results of the audio-visual capture of the real-world event. This envelops access to the output of the live editing task pertaining to the real-world event (i.e., the live broadcast signals that are generated by the ICoSOLE content producer(s)). Furthermore, metadata about the involved event is also required as input. Such metadata could, for example, be exploited to grant the consumer an impression of the physical layout of the spatially outspread event. Outputs List of the output of the service A hopefully satisfying and immersive consumer experience in which the end-user is afforded a comprehensive experience of the spatially dispersed event in real time (i.e., as it is unwinding at this very moment) and at a high audio-visual fidelity. Flow Description of the event flow when the service is activated 1. (A subset of) the live broadcast streams are delivered in real time to the consumer. 2. The consumer accesses the live broadcast(s) on his TV screen by tuning in to the appropriate "channel". 3. The consumer interacts with the live broadcast(s) via a simple, intuitive Graphical User Interface (GUI), which he controls via an off-the-shelf input device that is suitable for the TV set (e.g., a remote control). 3.1. The GUI allows the consumer to switch between (the live media streams that correspond with) the different locations/stages that constitute the spatially outspread event. 3.2. The GUI allows the consumer to install multiple views on the same content, for example through split screen functionality. 3.3. The GUI allows the consumer to select different audio playback styles and to perform advanced audio management (e.g., control the level of crowd noise). 4. The consumer ends his session by switching to another TV "channel" or by shutting down his TV set. Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)The act of interfacing with the service will be strictly reserved for end-users (i.e., no API that can be exploited by external software components is envisioned). * A GUI will allow users to provide input to the service in order to control the media presentation (e.g., content/sub-event selection, view and perspective selection, navigation between spatially dispersed yet temporally parallel sub-events, and so on). A low-complexity GUI is advocated, given the interaction constraints that are imposed by typical TV control equipment (i.e., remote controls). * It is envisaged that the operation of the service will be customizable on the basis of the end-user's media consumption preferences (e.g., making trade-offs between the dissemination of audio and video content when downstream bandwidth is scarce). These configuration settings might also be considered as a form of input to the service. Required resources Description of (external) resources involved * The live editing platform that is responsible for producing the live broadcast signals. * Metadata, including (but potentially not limited to) a model of the spatially outspread event and an event schedule including biographical information of the involved performers. * If necessary, a live broadcasting environment will be simulated by resorting to static, file-stored footage that is streamed in a fashion similar to live content. This need will arise when interfacing with the live editing component turns out to be problematic (e.g., during the development phase when both the live editing tool and the TV playout client are still actively being developed). Dependencies Software, hardware and platform dependencies * The delivery of the elementary streams will preferably be MPEG-DASH-based in order to maintain uniformity and consistency with the other playout-related ICoSOLE services. However, in case the



deployment platform (i.e., the TV set) would lack MPEG-DASH support, a fallback will need to be made to a streaming technology that is actually supported by the involved platform and which ideally exhibits a feature set similar to MPEG-DASH. * As the content dissemination will be IP-based, this service depends on the presence of IP networking infrastructure. * The playout client will most likely not be implemented as a native TV or set-top box application (due to the primarily closed nature of such hardware). An HTML5-based implementation that is accessible via the built-in web browser of Internet-connected smart TVs seems like a more feasible approach. Another option would be to resort to some sort of "TV plug-in peripheral" technology like an HDMI dongle running Android in order to "run" the playout client on the TV screen. Related information Documents, links to the service

7.5.4 Content pre-caching to improve broadcast channel zapping

Contributing partners iMinds Applies to use cases 4.1 Type Audience

Summary Brief overview of the requirements findings Digital TV distribution suffers from considerable latency issues when viewers attempt to switch between live broadcast signals (so-called "zapping"). In case the temporal overhead of zapping becomes too large, the risk arises that the viewer loses his interest in the content or, even worse, gets frustrated with the system. The number of concurrent broadcast signals in the ICoSOLE environment is likely to be high (e.g., federated real-world events involving various sub-events that are occurring in parallel, different views on the same sub-event, etcetera). Therefore, provisions are required to maximize the zapping speed. To this end, intelligent pre-caching strategies will be integrated in the playout client for live playback on TV. The pre-caching service will support heterogeneous types of video content (e.g., traditional video, omni-directional or panoramic) and the distribution of the elementary streams will preferably be MPEG-DASH-compliant. Scope & purpose What is to be achieved with this service? What is needed? By exploiting intelligent pre-fetching and pre-caching strategies, the time-to-first-picture when zapping from one live broadcast stream to another will be optimized. The goal is to support near-instantaneous (i.e., fast and uninterrupted) switching between broadcast streams in the playout client for live playback on TV. Inputs List of the required and optional input for the service * The service requires access to the results of the audio-visual capture of the real-world event. This envelops access to the output of the live editing task pertaining to the real-world event (i.e., the live broadcast signals that are generated by the ICoSOLE content producer(s)). * Metadata about the event and its involved elementary streams might also be required as input, as such information is likely to offer opportunities to optimize the pre-caching efficiency. * Consumer preferences might aid the pre-caching service at functioning optimally. These preferences can be highly heterogeneous in nature. For example, in case the service is informed of the fact that the viewer at hand prefers wide-angle views compared to close-up shots, the service might favour the former content type by proportionally allocating more resources for its pre-caching. * A history of previous consumption-related actions taken by the viewer. This input category is highly related to the previous one and might similarly be exploited to improve the pre-caching efficiency. In effect, the pre-caching service might be able to automatically "learn" consumer preferences on the basis of the actions the consumer has taken in the past, and subsequently exploit this knowledge to optimize its performance. Outputs List of the output of the service Live broadcast streams are pre-fetched and cached at client side to enable fast channel zapping. The pre-caching implies that (a subset of) the elementary streams are available locally before they actually become needed for rendering purposes. In other words, pre-cached content is not actually decoded nor rendered by the client until the consumer effectively switches to the corresponding channel. As at this stage the desired content is already available locally (potentially at an intermediate quality level



though), the channel switch will be near instantaneous. Pre-cached media are not decoded to preserve computational resources at client side. Flow Description of the event flow when the service is activated 1. The consumer connects to the ICoSOLE system on his TV set at home to consume the professional capture of the spatially outspread live event that is broadcasted in real time. 2. Alongside the delivery of the broadcast signal that the consumer is currently watching, a limited number of additional live streams pertaining to the event are pre-cached at client side. The pre-caching potentially occurs in degraded audio-visual quality, depending on the prevailing network conditions (e.g., available downstream bandwidth). 3. Whenever the viewer decides to switch to another broadcast stream: 3.1 In case the target signal is currently being pre-cached, the switch will be extremely fast. In case the pre-caching did not occur in optimal audio-visual quality, this quality will (either gradually or abruptly) be improved as the new stream is being rendered. 3.2 In case the target signal is not currently being pre-cached, the distribution platform will need to start streaming the corresponding elementary content to the client. This will result in a substantially higher time-to-first-picture for the target broadcast stream in comparison to situation 3.1. 3.3 In both cases, the pre-caching service records the viewer's action in order to update its internally maintained profile of the viewer (which includes consumption preferences). Subsequently, the pre-caching service applies its re-compiled knowledge base and the modified contextual information in order to decide on the set of broadcast signals that from this point on need to be pre-cached. 4. When the consumer ends his session by switching to a non-ICoSOLE TV channel or by shutting down his TV set, the pre-cache service is terminated and any pre-cached footage is discarded. Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)No externally accessible interface is envisioned for this service. Required resources Description of (external) resources involved * The IP-based network infrastructure that will implement the content distribution. * The live editing platform that is responsible for producing the live broadcast signals. * Any metadata about the real-world event and its involved elementary streams that might unlock opportunities to optimize the pre-caching efficiency. * If necessary, a live broadcast environment will be simulated by resorting to static, file-stored footage that is streamed in a fashion similar to live content. This need will arise when interfacing with the live editing component turns out to be problematic (e.g., during the development phase when both the live editing tool and the TV playout client are still actively being developed). Dependencies Software, hardware and platform dependencies * The delivery of the elementary streams will preferably be MPEG-DASH-based in order to maintain uniformity and consistency with the other playout-related ICoSOLE services. However, in case the deployment platform (i.e., the TV set) would lack MPEG-DASH support, a fallback will need to be made to a streaming technology that is actually supported by the involved platform and which ideally exhibits a feature set similar to MPEG-DASH. * As the content dissemination will be IP-based, this service depends on the presence of IP networking infrastructure. * The pre-caching service will most likely not be implemented as a native TV or set-top box application (due to the primarily closed nature of such hardware). An HTML5-based implementation that is accessible via the built-in web browser of Internet-connected smart TVs seems like a more feasible approach. Another option would be to resort to some sort of "TV plug-in peripheral" technology like an HDMI dongle running Android in order to "run" the pre-caching-enhanced live playout client on the TV screen. Related information Documents, links to the service

7.5.5 Venue Explorer

Contributing partners BBC



Applies to use cases 5.2c Type Audience

Summary Brief overview of the requirements findings The user (on a mobile platform) can navigate the available content for the event using a global map view or from a panoramic video of the live event. Graphical overlays and AV previews are used to aid navigation. Timeline navigation is also possible. The user can then select an AV feed for full screen consumption. Scope & purpose What is to be achieved with this service? What is needed? A spatial navigation interface for audience experience of the event on mobile platforms. Using a global scene view (map or video) with graphical overlays and AV previews, plus a timeline control. Inputs List of the required and optional input for the service The network streams representing the multimedia content User input in order to control the media presentation (e.g., pan/zoom navigation, content/stage selection, timeline control…) Metadata about the event and description of overlay graphics. Global event media (map/video) Outputs List of the output of the service Satisfied consumer Flow Description of the event flow when the service is activated 1. The consumer accesses the playout client on his/her mobile device and selects the Venue Explorer interface mode 2. A scene representation of available content is delivered to playout client including the spatiotemporal extent of each content item 3. Multimedia content is delivered to the consumer using DASH 4. The consumer interacts with the platform via the Venue Explorer interface a. The GUI presents the consumer with a global map-view of the event, indicating positions of available content items. b. The GUI allows the consumer to move back and forth on the timeline c. The GUI allows the consumer to pan/zoom around the global scene view d. The GUI allows the consumer to preview available content, and receive automatic picture-in-picture previews based on current viewport. e. The GUI allows the consumer to turn on and interact with overlay graphics 5. The consumer is able to select and view available content items in full-screen mode 6. The consumer ends his/her session Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST using JSON schema (for graphics description, scene model, event meta-data, user interactions)MPEG DASH delivery of media signals Touch screen navigation interaction Required resources Description of (external) resources involved DASH delivery of content, playout platform for users, binaural rendering Dependencies Software, hardware and platform dependencies Mobile device application with GUI, touchscreen. Related information Documents, links to the service

7.5.6 Control of TV stream selection

Contributing partners iMinds Applies to use cases 4.1, 4.2 Type Audience



Summary Brief overview of the requirements findings Content consumption in the ICoSOLE eco-system surpasses the classic notion of zapping between broadcast channels in traditional DVB systems. In effect, since the ICoSOLE project targets the immersive, multi-perspective capture of federated real-world events encompassing various sub-events that are occurring in parallel, the number of concurrent media streams will be high. The playout client for playback on TV consequently requires a service that empowers the consumer (i) to efficiently select the media signals in which he is interested, and (ii) to control their visualization on the screen. In order to enhance the immersive nature of the ICoSOLE experience, the service shall include advanced video presentation features, including side-by-side video layouts, picture-in-picture solutions, and/or effective tools to interact with omni-directional or panoramic visual footage. The service must offer a low-complexity GUI, given the interaction constraints that are imposed by typical TV control equipment (i.e., remote controls). To this end, a template-based approach to video stream visualization will be adopted. Scope & purpose What is to be achieved with this service? What is needed? The service allows the consumer to identify and select the desired ICoSOLE media content, and also empowers the consumer to decide on the layout of the selected video streams on the TV screen. The service can be applied to both the live and on-demand playback on TV use cases (i.e., use cases 4.1 and 4.2, respectively). The focus is on supporting both playback styles in a non-mixed fashion (i.e., the collection of media streams will exclusively consist of either live broadcast signals or on-demand content); controlling hybrid TV stream selection (involving live and on-demand content simultaneously) is considered an extension of the service. Inputs List of the required and optional input for the service * The service requires access to the results of the audio-visual capture of the real-world event. This envelops at least access to (i) the output of the live editing task pertaining to the involved event (i.e., the live broadcast signals that are generated by the ICoSOLE content producer(s)), and (ii) any relevant UGC that is registered with the ICoSOLE system. * Metadata about the involved event and the resulting broadcast/UGC media streams. Metadata could be exploited, for example, to organize and present the collection of available media signals in different ways, so that consumers at a glance can find the media stream they are looking for. For instance, based on geographic metadata pertaining to the spatially outspread event, the media signals could be pinpointed on a topographic map according to the real-world location to which they apply. Outputs List of the output of the service * A visualization of the list of involved media signals. The consumer is able to customize the list's representation by installing "filters" that are driven by metadata about the real-world event and its involved media streams. * Spatial layout and rendering of selected media signals on the TV screen. Flow Description of the event flow when the service is activated 1. The consumer "tunes in" to the ICoSOLE coverage of a real-world live event using the playout client for playback on TV. 2. The service presents to the consumer an overview of the media signals that pertain to the real-world event. 2.1 Via the use of metadata-powered filters, the consumer is able to tweak the presentation of this overview so that it becomes easier for him to perform the content selection. 3. From this overview, the consumer selects the media streams that he wants to visualize on the TV screen. 4. The consumer decides on a spatial layout for the selected media streams on the TV screen. 4.1 The consumer chooses between a predefined number of spatial layout templates. The templates list will include at least full-screen, side-by-side and picture-in-picture visualization options. 4.2 The consumer assigns each selected media stream to a spatial region that is defined by the selected layout template. 5. The selected media streams are rendered on the TV screen in the desired layout. 6. At this point, the user can consume the selected content, or he can return to step 2 in order to modify his media stream selection and/or the spatial ordering of the selected streams. 7. The consumer ends his session by "tuning out" or by shutting down his TV set.



Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)The act of interfacing with the service will be strictly reserved for end-users (i.e., no API that can be exploited by external software components is envisioned). * A GUI will allow users to provide input to the service in order to select a suitable subset of the entire collection of media signals that pertain to a particular event. * A GUI will allow users to decide on the spatial layout of the selected subset of media content on the TV screen. Required resources Description of (external) resources involved * The live editing platform that is responsible for producing the live broadcast signals (in the case of live playback). * The UGC repository pertaining to the real-world event (in the case of on-demand playback).* Metadata, including (but potentially not limited to) a model of the spatially outspread event and an event schedule including biographical information of the involved performers. * If necessary (for when dealing with live broadcasting use cases), a live broadcasting environment will be simulated by resorting to static, file-stored footage that is streamed in a fashion similar to live content. This need will arise when interfacing with the live editing component turns out to be problematic (e.g., during the development phase when both the live editing tool and the TV playout client are still actively being developed). Dependencies Software, hardware and platform dependencies * The availability of a suitable input device that allows the consumer to interact with the TV set. The specific focus is on off-the-shelf input devices in general and standard TV remote controls in particular.* The playout client will most likely not be implemented as a native TV or set-top box application (due to the primarily closed nature of such hardware). An HTML5-based implementation that is accessible via the built-in web browser of Internet-connected smart TVs seems like a more feasible approach. Another option would be to resort to some sort of "TV plug-in peripheral" technology like an HDMI dongle running Android in order to "run" the playout client on the TV screen. Related information Documents, links to the service

7.5.7 Moments Selection

Contributing partners VRT Applies to use cases 5.2b Type Audience

Summary Brief overview of the requirements findings This service generates a list of UGC videos (moments). This list can in turn be used to display in an end-user application. Scope & purpose What is to be achieved with this service? What is needed? Selection of moments based on specific criteria (recent, trending, etc.). Inputs List of the required and optional input for the service Metadata that is relevant to the selection of moments: social network information, calendar information, UGC information, location information, user profile information Outputs List of the output of the service A (sorted) list of moment videos and associated metadata Flow Description of the event flow when the service is activated 1. The service is activated through an external request to generate a list of moments 2. All necessary metadata is collected through various other services (social network info, calendar info, UGC info, location info and user profile info)



3. The filtered and ordered list of moments is returned Interface Interface & definition language used to communicate with the service (e.g. REST using JSON Schema)REST Required resources Description of (external) resources involved Metadata Dependencies Software, hardware and platform dependencies Needs information from services 1.1, 1.2, 1.6, 1.9, 1.11 and 5.2. Related information Documents, links to the service



8 Conclusions and Outlook

The ICoSOLE project intends to develop a system for immersive coverage of spatially distributed live events. The use cases, describing the interactions between users and the system, have been presented in this document in detail. These use cases were derived from the usage scenarios described in deliverable D2.1 and cover aspects of the production chain from planning through to audience experience. The use cases give a high-level overview of the proposed functionality and workflow of ICoSOLE. Innovative aspects of the system have been highlighted at this stage.

The use cases have also been prioritised. This is based on individual consortium partner’s priorities, considering where innovation can be delivered, where there are strong dependencies, and where there is most interest from broadcasters. This prioritisation will be used during the design of the system architecture and the system implementation, to allow the scope of the system to be appropriately limited and the available effort focussed on the most important aspects.

Further analysis of the use cases also led to a list of user interfaces. Though these user interfaces have not been described in detail, the identification of required use interfaces is crucial for a good interaction within the project.

A set of functional requirements has been identified from the use case descriptions, defining services required of the system to provide the functionality described in the use cases. Non-functional requirements have been identified during this process also. These requirements will provide more specific input to the system architecture and implementation design.

The service descriptions indicate the distinct functional blocks of the overall system that may need to be implemented and how they relate to the use cases. The high-level dependencies and interactions between services can be identified from this document. Many services are used by different use cases in parallel. For use cases that have high priority, activity charts have been created to aid this process. The results of this process, contained in this document, should make the system design more manageable and ensure that it is better structured, avoiding duplication and allowing flexibility.



9 References

[EBU, 2013] EBU Tech 3364, The Audio Definition Model



10 Glossary

Terms used within the ICoSOLE project, sorted alphabetically.

AirPlay Apple product to stream content from tablet to a TV, http://www.apple.com/uk/airplay

DASH Dynamic Adaptive Streaming over HTTP

FaceTime Apple’s video call technology, http://www.apple.com/uk/mac/facetime

Photo-Tourism Microsoft’s 3D photo system based on combining users’ photos at popular landmarks. http://research.microsoft.com/en-us/um/redmond/groups/ivm/PhotoTours

REST Representational State Transfer

Streetview Google’s street level photos, https://www.google.com/maps/views/streetview

UGC User Generated Content

Partner Acronyms

BBC British Broadcasting Corporation, UK

BIT Bitmovin GmbH, AT

DTO Technicolor, DE

iMinds iMinds Vzw, BE

JRS JOANNEUM RESEARCH Forschungsgesellschaft mbH, AT

TaW Tools at Work Hard+Soft Vertriebsgmbh, AT

VRT De Vlaamse Radio en Televisieomroeporganisatie NV, BE

Acknowledgement: The research leading to these results has received funding from the European Union's Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 610370.