Designing a Verification Tool forEasier Quality Assurance ofInteroperable Master FormatPackages

Martin Sjolund

VT20Master Thesis in Interaction Technology and DesignSupervisor: Ulrik SoderstromExternal Supervisor: Emil EdskarExaminer: Thomas MejtoftDepartment of Applied Physics and Electronics

i

Abstract

With today’s global distribution of movies, series, documentaries,and more, the need for a standardised system for storing contenthas emerged. Over-the-top media services such as Netflix, HBO,and Amazon Prime are storing large amounts of content, and byproviding it internationally, the content multiplies when it hasto conform to regional standards and regulations. The organisa-tion Society of Motion Picture and Television Engineers (SMPTE)has, in the light of this, created a standard called the Interoper-able Master Format (IMF). This component-based media lowersstorage costs drastically by only storing and managing the mediaelements that are unique between versions.

In management of media content, one of the tasks is verification, aprocess where the content is checked for errors. By incorporationthis process into an IMF workflow, the efficiency could be consid-erably improved. The objective of this thesis is to explore the useof IMF today and design a tool used for verification of IMF pack-age data, solving present problems in the verification workflow.By looking more deeply into the IMF standard and the needs ofpeople working with verification, a prototype could be created thatattends to the needs of the user while simultaneously conformingto the IMF workflow. The prototype was received well by designexperts and there is a potential of the further development of it.

Svensk Sammanfattning

Med dagens globala distribution av filmer, serier, dokumentarer,m.m., sa har behovet av ett standardiserat system for att lagrainnehall uppstatt. Over-the-top mediatjanster som Netflix, HBOoch Amazon Prime lagrar stora mangder innehall, och eftersom dear internationellt verksamma sa blir den mangden innehall mycketstorre nar det maste ratta sig efter de olika regionala standarderoch regler som finns. Organisationen Society of Motion Pictureand Television Engineers (SMPTE) har pa grund av det skapatstandarden Interoperable Master Format (IMF). Den standardengor innehallet komponentbaserad, vilket sanker lagringskostnaderdrastiskt genom att bara lagra och hantera media-element som arunika mellan versioner.

Verifikation ar en av uppgifterna nar man hanterar ljud och bild.Det ar en process dar man kollar igenom innehallet efter fel. Ef-fektiviteten av den processen skulle kunna bli betydligt forbattradgenom att inkorporera den i ett IMF-arbetsflode. Malet med dethar exjobbet ar att utforska dagens anvandning av IMF och sedandesigna ett verifikationsverktyg for IMF-paket. Syftet med verk-tyget ar att losa problem i verifikations-arbetsflodet. Genom attlasa mer om IMF och undersoka potentiella anvandare sa har en

ii

prototyp skapats som moter anvandarens behov samtidigt somden ar anpassad for ett IMF-arbetssatt. Prototypen mottogs valav designexperter och det finns potential for vidare utveckling avden.

iii

List of Abbreviations

CPL Composition Playlist.

IMF Interoperable Master Format.

IMP Interoperable Master Package.

OPL Output Profile List.

PKL Packing List.

QA Quality Assurance.

QC Quality Control.

SMPTE The Society of Motion Picture and Television Engineers.

UCD User Centered Design.

UI User Interface.

UX User Experience.

iv

Contents

1 Introduction 1

1.1 Objective 2

1.2 Codemill 2

1.3 Research questions 2

2 Background 3

2.1 The transition from 35mm film to digital video 3

2.2 Mastering of audiovisual content and the use of IMF 3

2.3 Quality control and quality assurance 4

3 Theory 5

3.1 Interoperable master format 5

3.1.1 Interoperable master package 5

3.1.2 Composition 6

3.1.3 Transcoding and distribution 7

3.2 User Experience 8

3.2.1 User centered design 9

4 Method 12

4.1 The general method approach 12

4.2 Literature study 13

4.3 Expert interview 13

4.4 Remote workshop 14

4.4.1 Workshop agenda 14

4.4.2 Workshop tools 15

4.5 Design iteration one - hand-drawn sketches 15

4.6 Design iteration two - digital wireframes 16

4.6.1 Expert evaluation of digital wireframes 16

4.7 Design iteration three - final prototype 16

v

5 Results 18

5.1 Summary of the expert interview 18

5.2 Results of the remote workshop 18

5.2.1 Analysis 20

5.3 “How Might We”-statements 20

5.4 Results of design iteration one - hand-drawn sketches 21

5.4.1 Analysis of design iteration one 21

5.5 Results of design iteration two - digital wireframes 22

5.5.1 Results of the expert evaluation 25

5.5.2 Analysis of design iteration two 26

5.6 Results of design iteration three - final prototype 26

6 Discussion 31

6.1 Discussion of results 31

6.2 Future work 32

6.2.1 The prototype 32

6.3 Testing methodology 33

6.4 The Coronavirus pandemic and changes to the methodology 33

6.5 Difficult interview with SVT 34

7 Conclusion 35

8 Acknowledgements 36

vi

1 Introduction

Cameras used for cinematography has for over 100 years been using the 35mm filmstock for capturing the frames that make up a movie or TV show [1]. Even thoughthe image quality of the 35mm film is of no concern, the capture and distributionof movies and series has in the 2010s become mostly digital. The transition to dig-ital has resulted in a shift in how we handle finished content, from rolls of film tofiles on a hard drive. Considering how sizeable high-definition footage can become,this can be challenging to file management systems. Especially today, when moviesand TV-series are distributed to so many different platforms and territories world-wide, destinations that require various adaptations such as dubbing of language, orrestrictions of explicit content.

The need for a standardized specification for file-based workflows arose with thedifficult task of managing all of these versions and files. This need gave birth to thestandard Interoperable Master Format (IMF), used for managing and processingmultiple content versions, created by the Society of Motion Picture and TelevisionEngineers (SMPTE)1 [2]. This standard made it possible to re-use common footageand audio between versions, thus saving a significant amount of storage space andreducing time spent on quality management. IMF allows for a master version ofa show to be created and stored, and when a different version is needed, only thedifference between the new version and the master is added to the file system [3].This is made possible mainly by the Composition Play List (CPL), an XML file withinstructions on how to combine the available video and audio elements that makeup a version [4].

IMF is used for internal and business-to-business relations and not intended to beused to deliver directly to the consumer. That means that a production companycan package its finished content and any existing versions in an IMF package andsend it to e.g. Netflix, who will ingest the content and process it into files suitable forstreaming to the consumer [5]. However, before sending an IMF package of finishedcontent, a Quality Assurance (QA) needs to be performed. The QA verifies thecontent of the IMF package, assuring it contains the correct files, and no errors hasoccurred to the files [6]. This paper aims to make that process more efficient bydesigning a tool for the professionals working with that verification process. Thedesign process will consist of a literature review, interviews with professionals anda workshop with experts in the field, identifying the requirements of the tool.

1https://www.smpte.org

1

1.1 Objective

The objective of this thesis is to explore the use of IMF today and design a toolused for verification of IMF package data, solving present problems in the verificationworkflow.

1.2 Codemill

This master thesis is written in collaboration with Codemill2 at their office in Umeaduring the spring of 2020. Codemill is a Swedish IT consulting firm founded in2007, experts in technical video solutions. With colleagues having MSc degrees incomputer science or interaction technology and design, they work with major inter-national companies, mostly in the media and broadcasting industry. They aim tokeep building their footprints across the media and broadcast industry internation-ally while keeping the heart and head office in Umea.

1.3 Research questions

This master thesis will explore the field of IMF, used in the business of video con-tent production and distribution. With the knowledge gathered, a tool used for theverification and quality assurance of IMF packages will be designed with the helpof UX methodologies and principles. To make this possible, information about IMFneeds to be acquired from a technical standpoint but also from the point-of-view ofprofessionals in the field. To assist the gathering of information and the master the-sis as a whole, some research questions have been composed with the main questionfollowed by a couple of sub-questions.

How would a tool used for verification of IMF packages be designed tosolve present issues while verifying content?

• What is IMF, and how is it used today?

• How is verification of IMF packages performed today?

• Who is performing quality assurance and would be the user of a QA-tool?

• What current issues are people working with verification facing today?

2https://www.codemill.se

2

2 Background

This chapter will cover some brief history of video production to then narrow itdown to the process of quality management. The purpose is to give some contextto the task of verification and the need for the Interoperable Master Format (IMF)standard in the field of video production.

2.1 The transition from 35mm film to digital video

About a decade ago, the majority of image capture in movie productions was doneusing a 35mm gauge [1]. 35mm gauge is a film format used for capturing and project-ing movies, documentaries, series, etc. The process of either capturing or projectingrequires several rolls of the plastic film [7]. However, in the last decade, the conver-sion to digital screens in cinema has made film stock a minority format [8]. Eventhough the opinion is split between which format is better video quality-wise [9], itis difficult to compete against the simplicity of the digital workflow versus handlinga large number of rolls in film stock.

2.2 Mastering of audiovisual content and the use of IMF

Audiovisual content today, such as movies and TV-series, is distributed to manydifferent territories and devices [10]. Because of this, the content must comply to thedifferent rules and regulations of those territories and the technical constraints of thevarious devices [11]. This is done with the concept of mastering. When producing apiece of content, e.g. a movie, a master version is created. This version is the originaland contains the director’s artistic intention, not affected by external restraints. Thismaster later needs to be altered to fit the regulations of an area, maybe censoring ofprofanity or suggestive content, or dubbed to the local language. The master mustalso conform to the different technical requirements of video and audio, depending onwhich devices the content is supposed to be consumed. For example, a 1080p versionfor BluRay, a 4K version for Netflix streaming, and other codecs and formats forvarious end destinations [12]. All these requirements lead to a large number ofversions of the original master, a phenomenon at Netflix, which they have chosen tocall “Versionitis” [13].

Storing all these masters and versions require a lot of storage; however, the useof an IMF (see section 3.1) workflow can create dramatic savings in storage andfile transfer. This is possible because IMF allows for the storage system to onlyhouse the content that is unique between versions. This storage system avoids the

3

replication of the same content which, other than saves on storage space, simplifiescontent tracking and content quality management [14].

2.3 Quality control and quality assurance

Quality control (QC) and quality assurance (QA) are components in the qualitymanagement process that exist to ensure the quality of a product. QC seeks toidentify and correct defects in a product, whereas QA tries to prevent defects bylooking into the making of the product [6]. Validation is an example of QC whileverification is an example of QA and could be defined as [15]:

Validation: “The assurance that a product, service, or system meets the needsof the customer and other identified stakeholders. It often involves acceptance andsuitability with external customers. Contrast with verification.”

Verification: “The evaluation of whether or not a product, service, or systemcomplies with a regulation, requirement, specification, or imposed condition. It isoften an internal process. Contrast with validation.”

This thesis will look more into the QA and verification process where the task is toassure the quality of video, audio and metadata files in an IMF package after theprocess of QC.

4

3 Theory

This chapter will contain the results of an extensive literature study conducted aimedto grasp the technical and practical aspects of the Interoperable Master Formatstandard, as well as the process of the User Experience design methodology.

3.1 Interoperable master format

Interoperable Master Format (IMF) is a standard created by the Society of MotionPicture and Television Engineers (SMPTE) used for the mastering and interchangeof file-based, multi version, audiovisual content [14]. Or:

“The IMF is a file-based framework that allows these high-quality ver-sions, called Compositions, to be efficiently represented, managed, playedback, processed, and transformed on file-based systems.” [16]

3.1.1 Interoperable master package

For delivery of content between businesses, the unit Interoperable Master Package(IMP) is used. [14, p. 8] It contains a Packing List (PLK), which is an XML filelisting all elements contained in the IMP [4, p. 144], along with video, audio, andmetadata elements. An IMP can be complete or partial. A complete IMP containsall files necessary for one or more compositions, and a partial IMP does not containall files necessary for one or more compositions. A practical use for this is if a newversion is to be sent and the recipient already has all the files for the original version,only the difference between the versions needs to be sent with a partial IMP [14,p.9].

The files contained by the IMP are essence data wrapped in MXF containers, andmetadata written in XML files. An IMP will at least contain the following [14, p. 8]:

• Packing List (PKL) XML file

• Assetmap (ASSETMAP) XML file

• Essence MXF file(s)

• Composition Play List (CPL) XML file(s)

5

Figure 3.1: An Interoperable Master Package (IMP) containing MXF and XMLfiles.

3.1.2 Composition

A version of some content in IMF is called composition and is defined by a uniqueComposition Play List (CPL) and the track files it references [13]. So, e.g. a movie’soriginal composition (version) is defined by a CPL and the original video and audiotrack. The German version is instead defined by another CPL but the same originalvideo track. The German composition may also contain a German opening videotrack and a German audio dub that replaces the original audio track.

The CPL is an XML file that contains references to essence track files and instruc-tions on how to combine them to create a composition.

6

Figure 3.2: A composition where the CPL is referencing the track files.

3.1.3 Transcoding and distribution

Video content is not only consumed on television screens, and there are severaldifferent video output devices, i.e., mobile phones, airline entertainment systems, andBlu-ray discs. Each is having different transcoding requirements. The transcodinginstructions, such as image cropping and colour transforms, can be stored in XMLfiles called Output Profile Lists (OPLs). They come contained in the IMP, and therecan be one or more OPLs per composition; however, an OPL is only referencing oneCPL. For example, one OPL may contain instructions for the Blu-ray transcodingof a composition, and another OPL may contain transcoding instructions for HDRTV transmission of the same composition.

7

Figure 3.3: The process of transcoding composition to different outputs using anOPL processor.

3.2 User Experience

User experience (UX) design is a process used to create products that provide rel-evant and meaningful experiences to the customer by including all aspects of theproduct from the User Interface (UI) to marketing and branding [17]. The totaluser experience distinguishes itself from just being the UI, even though it is alsoan important part. To have a good user experience the product must also, otherthan having a pleasant UI, deliver in the areas of branding, usability, design, andfunction. Alternatively, as the Nielsen Norman Group puts it:

“User experience encompasses all aspects of the end-user’s interactionwith the company, its services, and its products.” [18]

The seven facets of UX by Peter Morville [19] has also been widely adapted andcited as a framework for UX design [20]. It is also used to describe UX as more thanjust usability (see figure 3.4).

8

Figure 3.4: The UX honeycomb by Peter Morville [19].

The seven facets are described below with the help of Macpherson [21].

Useful Does the product, service or feature serve a purpose for your target cus-tomers?

Usable Is the user able to effectively and efficiently achieve their end goal?

Desirable The branding, image, identity and other elements of emotional de-sign [22].

Findable The ease of naviagation. Is the information the user seeks easy to find?Is the navigational structure intuitive?

Accessible The product or service should be accessible to users with disabilities.

Credible The ability of the user to trust in the product or service provided.

Valuable The product or service must deliver value to both the business and theuser.

3.2.1 User centered design

User centered design (UCD) is an iterative design process that looks for ways tosatisfy the needs of the user with the thing being designed (e.g. a product orservice) [23]. The process involves the user by various research and design techniques

9

such as interviews, usability testing, and observations [24].

The Double DiamondOne way to look at the design process is with the visual representation of it calledthe Double Diamond (figure 3.5). The process is divided into two diamonds. Thefirst is one of exploration where an issue is looked into widely or deeply. The seconddiamond is of taking focused action and delivering based on what is found outduring the first diamond. The design process is also not a linear one. Discoveriesin the exploration phase or testing of solutions may send the researcher back tothe beginning, where the process is reiterated for improvement. The diamonds aredivided into two sections each which tell more about the process.

Discover is about researching the issue and understanding the problem, rather thanassuming it. This is done by speaking to and spending time with the peopleaffected by the issues.

Define is about defining the problem in a new way from the insight gathered fromthe discover phase.

Develop is through collaboration with a wide range of people seeking differentanswers to the clearly defined problem. This is often started with some sort ofbrainstorm to gather many ideas.

Deliver is the further improvement or rejection of a solution whether it passessmall-scale testing or not. The testing will determine if the solution will workor not and is performed on an incrementally increased number of people basedon how close to the finished product you are.

10

Discover Define Develop Deli

ver

Connecting the dots and building relationships between different citizens, stakeholders and partners.

Creating the conditions that allow innovation, including culture change, skills and mindset.

OUTCOME

DESIGN PRINCIPLES1. Be People Centred

2. Communicate (Visually & Inclusively)3. Collaborate & Co-Create

4. Iterate, Iterate, Iterate

METHODS BANK

Explore, Shape, Build

CHALLENGE

© Design Council 2019

Figure 3.5: Design Council’s Framework for Innovation, 2019 [25].

11

4 Method

This section will describe the method this thesis has deployed to get its final re-sult. First, the general method approach is explained, then the specific methods aredescribed further in their own sections. Chapter 5 will follow a similar layout likethis one, which should enable the reader to jump between the two to see the resultsof previous steps and the decisions made before continuing with the next step (seefigure 4.1).

Figure 4.1: Visualisation of recommended reading order.

4.1 The general method approach

The method of this thesis is based on the user-centered design process and theDesign Council’s Framework for Innovation (see section 3.2.1). That means thatthe design process is divided into four main sections. The methods used in eachsection are chosen because of their user-centered approach. They are designed tocreate the solution that is best for the user.

DiscoverIn this case, information about IMF and verification is gathered through literature

12

studies. Also, an understanding is achieved of who the users are, what their tasks areand what issues they face performing their tasks. This understanding was achievedby an interview with a professional, and conducting a workshop with experts in thefield.

DefineHere, the problem(s) was defined using “How Might We”-statements [26].

DevelopUsing the “How Might We”-statements from before, solutions to the problem iscreated in form of prototypes that are later tested in the Deliver-phase. Theseprototypes were wireframes created using pen and paper, Balsamiq Wireframes1

and Sketch2.

DeliverHere, testing of the wireframes was performed by conducting expert evaluations. Thestep between Develop and Delivery is iterative and is performed until a desiredresult is achieved.

4.2 Literature study

By performing a literature study, a better understanding of the field could beachieved. The aim of the literature study was mainly to get a deeper understandingof the Interoperable Master Format (IMF) by reading technical specifications and re-ports. The aim was also to look into current research in the field of User Experience(UX) and incorporate it into the report and design process. Relevant research andtechnical specifications have been predominantly found using the following sources.

1. SMPTE3

2. DPP4

3. Google Scholar5

4. Diva-Portal6

4.3 Expert interview

To get some real information on how people work with verification, an interviewwas conducted with a person working at SVT7 in the TV production department.

1https://www.balsamiq.com2https://www.sketch.com3https://www.smpte.org4https://www.thedpp.com/5https://www.scholar.google.com6https://www.diva-portal.org7https://www.svtplay.se

13

The interview was an exploratory and qualitative one. The questions were more ofdiscussion points that we could talk over, i.e., “how does the verification processfunction at SVT?” or “Who are the people working with verification at SVT?”. Thisapproach to interviewing is called “semi-structured interviews” [27]. The goal of suchan interview is to gather information about a set of central topics while also allowingexploration when new topics present themselves. This approach was used because itwas the first real look into the field, and thus, limited knowledge had been attainedprior, making it too difficult to prepare more specific interview questions.

4.4 Remote workshop

To get a rough understanding of who the users of the verification tool are, a work-shop [28] was put together with experienced participants from Codemill. The choiceof conducting a workshop was because only one interview was managed to be done,and more information about the users and their tasks needed to be acquired. Thegoal of the workshop was to identify who the users could be and what kind of prob-lems they are and could be facing when working. For the workshop to be effective,a couple of things needed to be planned.

The objective of the workshopWith a predefined goal for the workshop, it becomes much easier to plan the activitiesand invite suitable participants. The goal for this workshop was, with the help ofthe Codemill employees define a persona of a user of a verification tool within thevideo production business. The use of that persona will be to formulate a use casewhich to make a design proposal about.

Which participants to inviteAll participants were from Codemill, and to get as much out of the workshop aspossible, the participants were invited based on their role in the company. Twodesigners and two salespeople were invited.

How the activities where designedWith the workshop goal in focus, the workshop activities were designed so the pre-vious activity helped the participants do the next activity.

4.4.1 Workshop agenda

With the main points above considered, an agenda of four activities for the workshopwas formed. Because the participants had worked with clients having employeesthat personas potentially could be based on, the first activity asked the participantsto think back on how their clients worked with verification and write some mainpoints on sticky notes. This activity was to give the participants a location where tofind someone for the second activity where participants were asked to describe thepeople working with verification, using a simple persona template they could putsticky notes on. The results of this activity would be used for the assembly of a userpersona later on.

14

After the first two activities, the participants now had one or more places and peopleon their minds, which would make the third activity more approachable. The thirdactivity asked the participants to think of potential issues and obstacles a userwould encounter while verifying content. Sticky notes would also be used here forparticipants to share their thoughts.

The last activity focused on getting the participants to come up with ideas forsolutions to the problems found in the previous activity. This activity was done bysplitting the participants into groups of two where whey could discuss solutions andideas together and later on present them to everyone in the workshop.

Main activity questions:1. How did previous or current clients of yours work with verification?2. Who are the people working with verification at these companies?3. What issues are these workers facing while performing their tasks?4. What does a verification tool need to resolve these issues?

4.4.2 Workshop tools

There existed some requirements of the tools used in the workshop because it wasbeing performed remotely with colleagues participating from home on their com-puter. Working remotely meant there was no office space in which to communicateand no whiteboard to collect ideas on. To overcome this obstacle, two tools wereused to digitally emulate a physical meeting.

Zoom8

Zoom is a video conference tool that would be used for communication by webcamera, microphone, and screen share. The tool also has a functionality where youcan create breakout rooms where you can send participants to discuss in smallergroups. The breakout rooms would become very useful for the last activity.

Miro9

An online collaborative whiteboard with much functionality, e.g. frames, stickynotes, and timers. This tool would be the centre of the workshop. Every activity hadits allocated area on the Miro-board. By screen sharing in Zoom, the workshop facil-itator could communicate to all participants how to continue with the next activity,and participants could share their screen and present their ideas.

4.5 Design iteration one - hand-drawn sketches

With the results of the workshop summarised, simple hand-drawn sketches could bemade. These sketches would be used as an early concept proposal to show the thesissupervisor at Codemill. The use of hand-drawn sketches was chosen because of theease of iteration when creating them. Additionally, according to Walker et al. [29],

8https://www.zoom.us9https://www.miro.com

15

there are few differences in feedback given when comparing low and high-fidelitysketches on paper or monitor.

With the approval of the direction of the concept and some feedback given, it wouldbe possible to continue with a second iteration.

4.6 Design iteration two - digital wireframes

To quickly test different versions of the same idea, the next iteration used a digitaltool to create wireframes from the original sketch of iteration one. This allowed foreasy modification of design elements without having to redraw all other parts.

The tool used was Balsamiq Wireframes. The tool consists of pre-made elementsdrawn in a low-fidelity fashion, which made it easier to focus more on layout andhierarchy than the aesthetic finish of the prototype. The purpose of the wireframeswhere to have an expert evaluate them before starting with the third and finaliteration.

4.6.1 Expert evaluation of digital wireframes

To test the validity of the wireframes, an expert evaluation was deployed by gettingfeedback from four design experts at Codemill. The amount of people evaluatingthe wireframes where decided because Nielsen et al. [30] states that when having upto five evaluations the return in evaluation results are growing rapidly but reachesthe point of diminishing returns when having about ten evaluators.

The evaluation process was that the facilitator explained the use case of which thewireframes are based and then let the evaluator look at the wireframes, ask questionsif needed, and give input on things that are good and things that could be improved.The input from the evaluators was noted and considered for the final iteration.

4.7 Design iteration three - final prototype

With the feedback received from the expert evaluations (see section 5.5.1) a morerefined prototype could be made using the vector-based design tool Sketch. Thisiteration would build on previous iterations and design solutions to the weaknessesfound during the evaluations. The use of Sketch also enabled the creation of clickableprototypes, making it easier to understand the flow of the design and spot potentialdesign flaws.

The process involved replicating the Balsamiq wireframes in Sketch, then tryingto solve the design flaws. Sketch was used because it made it possible to alter thewireframes in detail. While Balsamiq enables quick creation of wireframes, the useris only limited to pre-made elements. In Sketch, the user can create everything from

16

scratch; however, it requires more time.

17

5 Results

This chapter covers the results of the thesis. As mentioned in chapter 4, the layoutof this chapter will follow a similar layout as the method chapter allowing for thereader to jump between and follow the decisions made before continuing with thenext iteration.

5.1 Summary of the expert interview

The following are the main takeaways from the interview with SVT.

The person interviewed was a technical producer with several years of experienceat SVT. He explained how, when receiving material, such as an episode of a TVshow, it is sent to an in-house media centre. Before a Quality Control (QC) of thematerial can be performed, they check the codec of the files and transcode them totheir “house format”, which enables them to perform QC. The QC is a linear andmanual process where employees check the quality of the material in real-time usinga media composer, which is a very time-consuming task. The material is checkede.g. “loudness”, head and tail can be cut, and audio can be remapped. All this isdone before the material can be sent to their digital archive.

If there is something severely wrong with the material, SVT asks for a new deliverywith the problems fixed; however, if there is a minor error, it can be taken care ofby the SVT staff, i.e., lowering the volume of parts in the material.

The process of verifying the material is something SVT wants to look more into inthe future and possibly develop a better solution.

5.2 Results of the remote workshop

Below is summary of the remote workshop, divided into each of the four activities.The points summarised have been brought up multiple times during the workshopor have been discussed more thoroughly, motivating its relevance to the thesis.

What tasks are performed during verification• Goes through large lists of files to verify.• Verifies files by a large number of standards and checks that all components of adelivery is present and correct.• Review of metadata.

18

• Review of audio, video and subtitles.• Reporting of errors and issues.• May resolve small issues by themselves.• Mark files as approved or not.

Traits of a person working with verification• Some sort of education in media. Not necessarily a technical one.• 25-45 years old.• Working with repetitive tasks.• Uses a big screen (at least 24”), but sometimes a laptop.• Needs to minimise distractions.• Is thorough and determined.

Issues a person working with verification may face• Tight deadlines.• Large amounts of content to verify.• Collaboration can be difficult.• Software not working properly.• Working in old and outdated systems.• Functionality in the software is hidden behind “hot keys”.• Must work in several different systems.

Things that could help a person working with verification• Responsive software.• Software tailor made for the purpose of the task.• Customisable UI for specific users.• Self explaining UI, no hidden functions.• Make communication easier.• Automatic and manual verification in harmony.

Figure 5.1: The template used in the workshop for creating personas.

19

Figure 5.2: Participants filled the board with issues a person working with verifi-cation may face.

5.2.1 Analysis

Using the workshop summary, an adequate picture of a suitable persona, and a goodunderstanding of the day to day tasks a person working with verification has beenacquired. Going forward with prototyping, tasks such as reviewing audio and videoin a quality assurance manner will be explored. This means how quality assurancecan quickly verify the work done in the previous and more thorough task of qualitycontrol.

5.3 “How Might We”-statements

“How Might We”-statements could be created based on the results of the interviewand workshop (HWM = How Might We, QAW = Quality Assurance Workers):

• HMW make it easy for QAW to step between episodes/movies?• HMW make it easy for QAW to select parts of content to watch?• HMW make it easy for QAW to see if all the correct metadata is present?• HMW enable QAW to flag errors found in the content?• HMW make it possible for QAW to test subtitle timing?• HMW make it possible for QAW to check audio levels?• HMW make it possible for QAW to get an overview of the video quality?

20

5.4 Results of design iteration one - hand-drawn sketches

Because the focus of the thesis is to design a tool to be used by users who do quickfinal checks before sending or right after receiving content, the tool needed to enablethe user to go through the content of an IMF package easily. Because the contentof an IMF package is in itself difficult to comprehend only using a file explorer, thetool would have to help the user understand the compositions by using graphicalelements.

The idea of this design concept (shown in figure 5.3) was to enable the user toquickly review each composition (CPL) one at a time and colour code them de-pending on the status of the review. Also, each essence file is linearly shown on thetimeline with the purpose of clearly show how the composition is composed witheach track also being separated. The idea is that the user will be able to reviewthe tracks individually, and already reviewed tracks in previous compositions do notneed to be reviewed again.

Figure 5.3: First sketch of the verification tool.

5.4.1 Analysis of design iteration one

The concept was received well, and the decision to continue on that path was made.Some feedback was given, namely that the order of the UI elements should be re-considered to reflect their importance in the tool better. For example, the metadatafield can be closer to the CPL list.

21

Also, the use of a CPL list to go through as a user was a feature that should belooked more into going forward. It may prove to be useful.

5.5 Results of design iteration two - digital wireframes

The layout of the wireframes (figure 5.4-5.8) is based on the first sketch, which issimilar to the design of a traditional media composer application which is familiarto the people working with verification, as mentioned by the expert interview insection 5.1. The wireframes are made around the use case that the user has receiveda package of episodes from a TV show and is supposed to verify that the qualitycontrol has been done right. The timeline is supposed to change depending onwhich composition the user chooses in the composition queue (top left in figure 5.4and top row in figure 5.5-5.8) and the purpose of the timeline is to visualise the“stitching together” of video and audio files. Additionally, the fields “Essence trackmetadata” and “Audio levels” are information deemed important according tothe workshop (see section 5.2) and expert interview, and the information displayedin those fields are supposed to be the information of the tracks that the play headis currently on.

Another feature (seen in figure 5.7) is the ability to mark tracks as approved ornot after the user has verified them. This is supposed to make it easier for the userto keep track of which tracks have been verified and which have not by changingthe colour of the track according to the decision. Because IMF can re-use tracksfrom another composition without duplicating it, the user should not need to verifyalready verified content. One thought is, therefore, to have the application rememberwhich tracks are verified already and have those tracks already colour coded on thetimeline, saving the user some time.

In summary, the user is supposed to be able to easily verify the sound and videoquality, the metadata, and subtitles of all compositions in an IMF package. Markthem as verified and not have to verify the same content more than once.

22

Figure 5.4: First version of the QA tool wireframe in Balsamiq.

Figure 5.5: Second version of the QA tool wireframe created in Balsamiq.

23

Figure 5.6: Third version of the QA tool wireframe created in Balsamiq.

Figure 5.7: Third version of the QA tool wireframe with right-click functionality.Marking as approved.

24

Figure 5.8: Third version of the QA tool wireframe with right-click functionality.Marking as “flawed”.

5.5.1 Results of the expert evaluation

This section will present the summarised results of the expert evaluations (see section4.6.1) conducted at Codemill.

The use of tabs (see figure 5.5-5.7) to keep track of compositions in the IMFpackage got mixed reviews leaning to it being understandable and preferred overthe file explorer alternative (see figure 5.4). However the size of the tabs could besmaller, giving more space to other elements.

The use of several metadata fields could make the interface messy while reviewingmetadata or moving a lot in the timeline. While the metadata field in itself wasunderstood by the evaluators, the number of metadata fields could be reduced fromtwo to one by default, and the metadata within the fields needs separating.

The user might want to go full screen with the video preview window or detach itcompletely from the timeline. This allows the user to see the video in full screen onone monitor and the timeline on another.

Using coloured bars in the timeline to visualise the IMF “stitching together” ofessence tracks worked well. Also, the concept of colour code the tracks depending ontheir status, was well received. However, an argument was made that when changingthe status of a bar (as seen in figure 5.7), the whole essence track should be seenas reviewed and change colour depending on the status. For example, if an error wasfound in the first part of the track, the entire file should be seen as defected because

25

it must still be replaced entirely.

The findability [31] of the “mark as approved/flawed”-function may be low consid-ering the high importance of the function.

Other remarks:1. The top file directory in the file explorer is redundant.2. The review of audio and video transitions may be interesting.3. You can not see the name of the IMF package being reviewed in the tool.4. The audio tracks may use more audio channels, requiring more audio levels to beverified.5. Having subtitles in the video player instead of in a separate field was preferred.6. There is no ability to see which track is currently active.7. Use the work “rejected” instead of “flawed”.8. The ability to mute tracks or audio all together could be useful.

5.5.2 Analysis of design iteration two

The feedback received from the expert evaluations proved themselves very valuable,and it was all to be addressed in the final design iteration. The point of marking theentire essence track instead of just a part of it was a major insight, as well as howthe current state (design iteration two) of the metadata fields could be confusing.

With the evaluations summarised, a more refined prototype could be created withchanges to combat the design weaknesses discussed with the evaluators.

5.6 Results of design iteration three - final prototype

Figure 5.9 to figure 5.14 shows the results of the final design iteration. Thesewireframes made in the design tool Sketch are the approved upon versions of thewireframes made in design iteration two (see section 5.5), based on the expertevaluations conducted (see section 5.5.1). The use case is still that the user is toverify an IMF package containing six compositions of a TV show. However, in thisuse case, the show is a dubbed version in German.

Following list specifies the changes made from previous iteration:1. The IMF package name is displayed in the top left of the window.2. The top file directory is removed from the file explorer to the left, due to redun-dancy.3. There is now only one metadata field and the placeholder metadata has beenadded for demonstration purposes.4. More audio levels have been added showing the possibility of the audio level field.5. The essence track can now be selected, showing which track is currently active(figure 5.10).6. The metadata inside the metadata field is changing depending on which track isselected, rather than where the play head is positioned.

26

7. The audio track in the audio levels field is changing depending on where the playhead is positioned.8. The verification options are now “Mark as approved” and “Mark as rejected”(figure 5.11).9. When marking a track as either approved or rejected, the whole track is nowcoloured instead of just the track block (figure 5.10 & 5.11).10. A check mark (approved) or a block sign (rejected) is added next to the trackfile name when marking tracks approved or rejected.11. When all tracks are approved, the colour of the composition tab is changed togreen (figure 5.14).

Figure 5.9

27

Figure 5.10

Figure 5.11

28

Figure 5.12

Figure 5.13

29

Figure 5.14

30

6 Discussion

6.1 Discussion of results

There are three things I would like to analyse to determine whether the results ofthe thesis are acceptable or not. Those three things are connected to the theory usedin this thesis:

1. Is the solution adapted for use of IMF packages?

2. How do Peter Morvilles seven facets of UX correspond to the results?

3. Has a User Centered Design process been used?

Is the solution adapted to use with IMF packages?The tool is designed to allow the user to go through an IMF package of CPLs as wellas verifying individual essence tracks connected to the CPLs. The tool also takesinto consideration that the essence tracks can have been previously verified. Thisplays on the great strength of IMF, the need to only store content that is uniquebetween versions of audiovisual content. By having those features, metadata andvisual aid to make compositions comprehensible, I would argue that the solution iswell adapted for verification of IMF packages.

How do Peter Morvilles seven facets of UX correspond to the results?Because the results are still in the prototype phase, some of Morvilles’s facets areredundant in this case. However, if the solution is Useful, Usable and Findable, arefacets worth discussing. I believe the usefulness of the solution is best describedin the paragraph above, furthermore, the purpose of the tool is well established inthe research phase. However, that the tool is useful does not mean that it is usable,because no user tests have been performed, it is difficult to deem the tool usablejust on the opinion from expert evaluations.

The findability of the “approve/reject” feature was briefly mentioned in the resultsection. There is a findability weakness to it because it is hidden behind a “right-click”, which means that new users can have difficulty finding it. The findability ofall features should be further looked into in future work.

Has a User Centered Design process been used?The Design Council’s Double Diamond has inspired the entire design process fromresearch to final prototype. The research was done to understand the users and theirissues as accurately as possible instead of using assumptions. The prototypes weremade to solve the issues found in the research phase, which are in extension, theusers’ issues. The weakness in this user centered design process was the lack of realusers to interview and test prototypes on, which forced adaptations to the method.

31

In conclusion for the last point, while the method was user centered, it could havebeen better if access to real users would have been made possible.

6.2 Future work

The final prototype for this thesis is finished; however, it does not mean that it isthe final solution for the tool. It just means that there is no more time to test andreiterate. If it was continued to be worked upon, three things could be done:

1. Test it on real users.2. Adapt it to the workflow of a client.3. Design it using Codemill’s design system1.

More valuable input could be gathered and used to improve the prototype by testingit on real people working with verification tasks in production. More importantly,the validity of the concept would also be tested to find out if the use case is aslegitimate as first thought. This ties to the second point of the list, adapting itto the workflow of the client. As companies have different workflows and routines,the validity of the concept may change. It may work better in some companiesthan others, depending on how their workflow aligns with the prototype. The usecase for this prototype is based upon the readings made on quality management ofsome companies, the technical specifications of the IMF standard, the results fromthe remote workshop, and the interview made with SVT. However, if one wouldcontinue designing and developing this tool for a client, an extensive study of thatclient’s workflow would have to be made to align the tool with their needs. As thereare many parties involved and systems used in the production of audiovisual media,it is almost impossible to have a solution that fits for all.

Lastly, when reiterating this prototype, effort should be put in to make it follow thedesign system. That way, if it were to be developed, much less effort would haveto be put in to make it harmonise with the rest of Codemill’s products than if theadaptation to the design system would be made afterwards.

6.2.1 The prototype

This section will discuss the further designing of the prototype.

One thing that does not exist in this prototype that would be a probable scenariowhile verifying content is the ability to approve the transitions between essencetracks. Even if the content of two tracks is correct without defects, the transitionbetween them may not be timed correctly. In that case, the possibility to rejectthe transition instead of the track would maybe be preferable because of the trackstechnically being correct. In this prototype, it is only possible to add comments tothe track verification; however, that function has not been further explored in this

1Suresh, Swetha, “Design Language System”, UX Planet (February 2018)

32

thesis.

The visibility of the “approve” and “reject” function could be further explored. Onecould argue that the function is hidden in its current state, and given the importanceof the task in this use case, the tool may benefit from having the function morefindable. One suggestion is to have it in the same row as the play and skip buttons,or having it next to the essence track name as a checkmark and “X”. The benefit ofhaving it as a “right-click” option is that it is more simple to describe the functionin text rather than symbols.

The file explorer field is in this prototype has not gone through any real thought offunction other than an overview of the IMF package. The field would much likelyalso be suitable for file selection. Instead of just being able to select files on thetimeline, the user would also be able to select files in the explorer to review. Maybealso being able to approve tracks directly from the file explorer.

More functions will have to be added to the video player, simple things such as mute,go full screen, and detaching the player. This would add to the overall legitimacyof the tool.

6.3 Testing methodology

The original plan was to contact companies with potential users of the tool andtest the prototype on them. This would probably have given some valuable insight.However, it proved difficult to reach such people, and a backup plan had to be de-ployed because of time constraints. The backup plan was to have design expertsat Codemill evaluate the prototype, experts who also have an insight into the ver-ification process because of the clients they have been working with. The feedbackreceived was regarded as valuable, even though the feedback from real users wouldhave been more valuable. However, because users may not be used to giving designfeedback, the amount of feedback may had been less if only users would have beeninvolved.

6.4 The Coronavirus pandemic and changes to the methodology

When this thesis was written during the spring of 2020, the world was undergoing apandemic caused by the Coronavirus2. This pandemic led to global measures beingtaken to prevent the rapid spread of the virus, one of which were the act of “socialdistancing” which led to many offices in Sweden started having their employeesworking from home, Codemill included.

Working from home meant some changes had to be made continuing with this thesis,also while some challenges presented themselves. The original plan was to interviewsome companies working with verification to gather information about users and

2https://www.who.int

33

workflows; however, getting a response proved itself difficult. Because of the diffi-culty of organising interviews and being past its time frame for the thesis, it wasdecided that a workshop was to be conducted instead. Instead of having the work-shop conducted at the office, it instead had to be done remotely using digital toolsbecause of the work from home recommendations. Having the workshop being per-formed remotely did not prove itself to be a significant disadvantage, compared toperforming it at the office; however, the planning of it is believed to have taken oneor two weeks longer.

Also, the feedback and expert evaluations had to be received via voice and videochat. Even though this worked, it was a bit more of a hassle organising the meetingsand creating the digital workspaces instead of meeting in an office.

The effect of the Coronavirus pandemic on this thesis could be summarised to havingto perform interviews, the workshop and evaluations remote, while also having thetime frame delayed a couple of weeks because of readjusting to working from homeand requiring more planning time for thesis tasks.

6.5 Difficult interview with SVT

I would like to add that the interview with SVT proved itself to be quite difficult.Because it was the first insight on a real verification workflow, the questions hadto be more discussion points for exploration of the verification area. The goal ofthe interview was to get more of an insight into the workflow, and who the peoplewere; however, the conversation steered more into technical aspects I had no to littleknowledge of. It would be interesting to talk with them again and ask some morespecific questions about their process and compare it to the knowledge I have gainedfrom last time.

34

7 Conclusion

The objective of this thesis was mainly to design a tool used for verification of IMFpackage data. That part of the objective is met with the prototype presented in theresults, evident by the tool being adapted for IMF, which is explained further insection 6.1. The other part of the objective is that the tool is to be designed to solvepresent problems in the verification workflow. The requirements for that part of thethesis is mostly met by research made on the verification workflow by interview andworkshop, however, user tests on the prototypes would have made the results morereliable.

Because of the user centered design approach that was used for this thesis, the mainresearch question formulated in the introduction could be answered in a legitimateway. Three of the four sub-questions could also be answered and helped in creatingthe final solution.

• What is IMF, and how is it used today?

• Who is performing quality assurance and would be the user of a QA-tool?

• What current issues are people working with verification facing today?

The second sub-question could not be answered because of the lack of access topeople working with the IMF. Instead, more general research about verification hadto be done, which gave adequate results.

• How is verification of IMF packages performed today?

35

8 Acknowledgements

I would like to thank the Codemill team for welcoming me to their office and helpingme throughout the thesis. A special thanks go out to my supervisor at Codemill,Emil Edskar, who guided me through several issues. Last but not least, I would liketo thank my dear friends Filip Bark, Fredrik Ostlund, Petter Poucette and OscarThorwid, who provided support and motivation during the thesis, as well as duringthe five years of university.

36

References

[1] Ivan Radford. Don’t keep it reel: why there’s life after 35mm. The Guardian,11 2011.

[2] Telestream. A guide to the interoperable master format (imf). http://www.telestream.net/pdfs/datasheets/App-brief-Vantage-IMF.pdf.Accessed: 2020-02-21.

[3] DPP. The interoperable mastering format.https://www.thedpp.com/imf/imf-overview. Accessed: 2020-06-10.

[4] Entertainment Technology Center. Interoperable master format (imf).Technical Report 1, Entertainment Technology Center, 509 West 29th Street,Los Angeles, California 90007, USA, 2 2011.

[5] Sreeram Chakrovorthy Rohit Puri, Andy Schuler. The netflix imf workflow.Netflix Tech Blog, 4 2016.

[6] Diffen.com. Quality assurance vs. quality control. https://www.diffen.com/difference/Quality_Assurance_vs_Quality_Control.Accessed: 2020-06-01.

[7] Matthew Wegenknecht. The actual costs of film.http://www.matthewwagenknecht.com/the-actual-costs-of-film/.Accessed: 2020-06-01.

[8] Leo Barraclough. Digital cinema conversion nears end game. Veriety, 6 2013.

[9] Caroline Siede. Maybe the war between digital and film isn’t a war at all. AVclub, 8 2018.

[10] Olsberg/SPI. Building sustainable film businesses: the challenges for industryand government, pages 27–30. Olsberg/SPI, 2012.

[11] CLAi.tv. Film, video internet mastering.https://www.clai.tv/film-video-internet-mastering/. [Accessed:2020-05-30].

[12] Charles S Swartz. Understanding digital cinema: a professional handbook,pages 83–115. Taylor & Francis, 2005.

[13] Sreeram Chakrovorthy Rohit Puri, Andy Schuler. Imf: A prescription forversionitis. Netflix Tech Blog, 3 2016.

37

[14] Digital Production Partnership. Imf operational guidance.https://cdn.digitalproductionpartnership.co.uk/wp-content/uploads/2019/07/DPP005-IMF-Operational-Guidance-2019-07-08.pdf.Accessed: 2020-02-18.

[15] Ieee draft guide: Adoption of the project management institute (pmi)standard: A guide to the project management body of knowledge (pmbokguide)-2008 (4th edition). IEEE P1490/D1, May 2011, pages 1–505, 2011.

[16] Dom Jackson. The interoperable mastering format.https://www.smpte.org/sites/default/files/section-files/BBTB2017-W04DominicJohnson-IMF.pdf. Accessed: 2020-02-19.

[17] Interaction Design Foundation. User experience (ux) design.https://www.interaction-design.org/literature/topics/ux-design.[Accessed: 2020-05-30].

[18] Jakob Nielsen Don Norman. The definition of user experience (ux).https://www.nngroup.com/articles/definition-user-experience/.[Accessed: 2020-05-30].

[19] Peter Morville. User experience design.http://semanticstudios.com/user_experience_design/, 2004. [Accessed:2020-05-30].

[20] Peter Morville. User experience honeycomb.https://intertwingled.org/user-experience-honeycomb/, 2016.[Accessed: 2020-05-30].

[21] Ellen Macpherson. The ux honeycomb: Seven essential considerations fordevelopers. Medium, 10 2019.

[22] Donald A Norman. Emotional design: Why we love (or hate) everyday things.Basic Civitas Books, 2004.

[23] Elizabeth B-N Sanders. From user-centered to participatory designapproaches. In Design and the social sciences, pages 18–25. CRC Press, 2002.

[24] Interaction Design Foundation. User centered design. https://www.interaction-design.org/literature/topics/user-centered-design.[Accessed: 2020-05-31].

[25] Design Council. What is the framework for innovation? design council’sevolved double diamond.https://www.designcouncil.org.uk/news-opinion/what-framework-innovation-design-councils-evolved-double-diamond,2019. [Accessed: 2020-03-04].

[26] Designkit.org. How might we. https://www.designkit.org/methods/3.[Accessed: 2020-07-05].

[27] Chauncey Wilson. Interview techniques for UX practitioners: A user-centereddesign method, pages 24–41. Newnes, 2013.

38

[28] Kate Kaplan. Ux workshops vs. meetings: What’s the difference? NielsenNorman Group, 2 2020.

[29] Miriam Walker, Leila Takayama, and James A Landay. High-fidelity orlow-fidelity, paper or computer? choosing attributes when testing webprototypes. In Proceedings of the human factors and ergonomics societyannual meeting, volume 46, pages 661–665. SAGE Publications Sage CA: LosAngeles, CA, 2002.

[30] Jakob Nielsen and Rolf Molich. Heuristic evaluation of user interfaces. InProceedings of the SIGCHI conference on Human factors in computingsystems, pages 249–256, 1990.

[31] Jen Cardello. Low findability and discoverability: Four testing methods toidentify the causes. Nielsen Norman Group, 9 2019.

39