embedded metadata in mxf - ebucore

Embedded metadata in MXFDieter Van Rijsselbergen

2012 EDM Metadata Workshop

EBU, Geneve – 20/06/2012

Who are we?

Contents

• Metadata exchanges?• Metadata in MXF• How to put metadata in MXF?• Putting EBUCore into MXF• EBU MXF SDK • EBU MXF SDK – Demo• EBU MXF SDK – Improvements & Community

Metadata exchanges?

• When exchanging audiovisual essence…– We must also exchange its identity,

its context, its creators,….– i.e., the semantics

behind the essence– in the form of

metadata

Metadata

Metadata exchanges?

• 2 ways to exchange metadata during the production and broadcasting process

• In-band /w essence– Tightly coupled and

doesn’t get lost– Often optimized for

essence storage

• Out-of-band– Centrally stored– Optimized for

metadata (search, augmentation, …)

Metadata exchanges?

• Out-of-band is interesting within the facility– Metadata is quickly accessible– Metadata can be indexed, annotated, …

• Embedded metadata is interesting in B2B– Side-car metadata doesn’t get lost,

doesn’t need a dedicated channel, …– E.g., news exchange scenarios• No more need for NewsML XML,

just serialize it directly into the MXF file

Contents

• Metadata exchanges?• Metadata in MXF• How to put metadata in MXF?• Putting EBUCore into MXF• EBU MXF SDK • EBU MXF SDK – Demo• EBU MXF SDK – Improvements & Community

Essence Exchange: MXF

• The Material Exchange Format is here to stay• Standardized by SMPTE and EBU• One of the most versatile, flexible and

powerful container formats defined– Not sure there’s anything it can’t do…– But makes it complex and harder to use

Metadata in MXF

• MXF has become the de-facto standard for audiovisual material production and exchange

• Fortunately, it has significant support for embedded metadata

Metadata in MXF

• 2 kinds of metadata in MXF– Structural metadata• Defines the structure of the file

– Track layout– Describes the essence,

which tracks, which codecs and parameters per track

– References to other/older files

Metadata in MXF

• 2 kinds of metadata in MXF– Descriptive metadata• Exchange of essence semantics

– e.g., a continuity report

• At its simplest, a means of identification– “let the descriptive metadata reference

identifiers I use internally in my organisation”

• This is the kinds we’re interested in!

Descriptive Metadata in MXF

• Standardized MXF descriptive metadata exists– Descriptive Metadata Scheme-1 (DMS-1)• Standardized as SMPTE 380M• Production characteristics, rights information,

annotations, person and entity identification, …• Aligned with MPEG-7, TV-Anytime

– Is DMS-1 being used for real?• Perhaps due to the fact that there is no DMS-1 notion

outside of the MXF ecosystem?– Hard to obtain a viable and complete mapping

Descriptive Metadata in MXF

• Which other descriptive metadata are interesting to consider?– General identification and description: EBUCore– Format-specific descriptions: IPTC NewsML-G2– Acquisition metadata (state of the recording

device): EBU Tech 3349– Feature detection results: AVDP profile of MPEG-7

Contents

• Metadata exchanges?• Metadata in MXF• Putting EBUCore into MXF• How to put metadata in MXF?• EBU MXF SDK • EBU MXF SDK – Demo• EBU MXF SDK – Improvements & Community

How to put metadata in MXF?

• Structure of MXF files is KLV elements– Key to identify the kind of element– Length of the value that follows– Value of the element

– KLV is for both essence and metadata• Typically, for essence a single frame per KLV

Key Length Value

16 bytes16 bytes BER-encoded length

BER-encoded length Length lengthLength length

......

How to put metadata in MXF?

How to put metadata in MXF?

• Use KLV to embed metadata<?xml version="1.0" encoding="UTF-8"?><ebuCoreMain xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:dc=http://purl.org/dc/elements/1.1/ xmlns="urn:ebu:metadata-schema:ebuCore_2011" xmlns:default=http://iptc.org/std/nar/2006-10-01/ xmlns:ebu=http://ebu.org/nar-extensions/ version="1.3" dateLastModified="2011-01-31"> <coreMetadata> <title> <dc:title xml:lang="fr-can">Berlin Wall 50th</dc:title> </title> <alternativeTitle typeLabel="headline"> <dc:title>Berlin Wall 50th</dc:title> </alternativeTitle> <creator entityId="EBUNMSSRVCodeSource:TBA"/> <subject> <dc:subject/> <subjectCode>EBUNMSCategoryCode:EXCH</subjectCode> </subject> <subject> <dc:subject>In or about Europe and EU</dc:subject> <subjectCode>EBUNMSWebcatCode:EUROPE</subjectCode> </subject> <description typeLabel="pubStatus"> <dc:description>NMSStatusCode:TBC</dc:description> </description> <description typeLink="EBUNMSDescription:dopesheet"> <dc:description> The Berlin Wall (German: Berliner Mauer) was a barrier constructed by the German Democratic Republic (GDR, East Germany) starting August 13, 1961, that completely cut off West Berlin from surrounding East Germany…

Key Length Value

16 bytes16 bytes BER-encoded length

BER-encoded length Length lengthLength length

......

Dump the document as a whole into the KLV value?Not the best approach…

How to put metadata in MXF?

• On top of this raw KLV structure is a conceptual model– Elements can be grouped into

logical metadata sets identified by a single key, i.e., “classes”

– Logical metadata sets can be reconstructed by means of a dictionary

<Identification base="InterchangeObject" detail="Identification set" type="localSet" baseline="yes" key="06 0e 2b 34 02 53 01 01 0d 01 01 01 01 01 30 00"> <ThisGenerationUID use="required" type="UUID" key="3c 09" globalKey="06 0e 2b 34 01 01 01 02 05 20 07 01 01 00 00 00"/> <CompanyName use="required" type="UTF16String" key="3c 01" globalKey="06 0e 2b 34 01 01 01 02 05 20 07 01 02 01 00 00"/> <ProductName use="required" type="UTF16String" key="3c 02" globalKey="06 0e 2b 34 01 01 01 02 05 20 07 01 03 01 00 00"/> <ProductVersion use="optional" type="ProductVersionType" key="3c 03" globalKey="06 0e 2b 34 01 01 01 02 05 20 07 01 04 00 00 00"/> <VersionString use="required" type="UTF16String" key="3c 04" globalKey="06 0e 2b 34 01 01 01 02 05 20 07 01 05 01 00 00"/> <ProductUID use="required" type="AUID" key="3c 05" globalKey="06 0e 2b 34 01 01 01 02 05 20 07 01 07 00 00 00"/> <ModificationDate detail use="required" type="Timestamp" key="3c 06" globalKey="06 0e 2b 34 01 01 01 02 07 02 01 10 02 03 00 00"/> <ToolkitVersion use="optional" type="ProductVersionType" key="3c 07" globalKey="06 0e 2b 34 01 01 01 02 05 20 07 01 0a 00 00 00"/> <Platform use="optional" type="UTF16String" key="3c 08" globalKey="06 0e 2b 34 01 01 01 02 05 20 07 01 06 01 00 00"/></Identification>

How to put metadata in MXF?

• On top of this raw KLV structure MXF definesa conceptual model– Metadata sets can reference other sets– Metadata set can be

constructed as trees or graphs

How to put metadata in MXF?

• Don’t just throw the metadata in there– Link the descriptive metadata

to the structural metadata

The DM Framework hooks custom descriptive

metadata to (a part of ) the MXF structural timeline

How to put metadata in MXF?

• KLV encoding + conceptual data model + timeline model

• MXF has a proper way of structuringmetadata natively– Whenever we can, we should use it!

Contents

• Metadata exchanges?• Material Exchange Format• How to put metadata in MXF?• Putting EBUCore into MXF• EBU MXF SDK • EBU MXF SDK – Demo• EBU MXF SDK – Improvements & Community

Putting EBUCore into MXF

• A case study in embedding MXF metadata and with practical merit– Embed an EBUCore XML document in MXF file

using a proper KLV mapping and encoding– Build software that handles

MXF container and its embedded EBUCore metadata as a whole

ITM504483_EBUcore.

XML

EBU MXF SDK

• Initiative by EBU to provide community with reliable and open-source software to manipulate MXF files– Handle metadata– Handle essence and work on

conformity of MXF files– Available as free and open source software• First repository: https://github.com/Limecraft/bmx

EBU MXF SDK

• Based on BBC librarieslibMXF, libMXF++ and bmx– Reference implementations for MXF app. specs.• AMWA AS-02 (Versioning) and • AMWA AS-11 (MXF for Contribution)

– Written in C/C++• Milestone #1– Mux and de-mux EBUCore metadata

in MXF containers

EBU MXF SDK – Demuxing/Reading

CompressedVideo

Frames

CompressedVideo

Frames

CompressedAudio

Samples

CompressedAudio

Samples

FFmpeg(libavcodec)

MXF SDK

ReadPartitions / Structural Metadata

ReadDescriptiveMetadata

ReadIndices

Read Essence

SMPTE 434 XML Representation

Metadata(EBU Core)

MXF AnalysisReport

CompressedAudio

Samples

CompressedVideo

FramesDecodeVideo

DecodeAudio

AudioSamples

VideoFrames

EBU Player/GUI

Playback

Metadata Display/Analysis

(Files)

EBU MXF SDK – Wrapping/EBUCore

CompressedVideo

Frames

CompressedVideo

Frames

CompressedAudio

Samples

CompressedAudio

Samples

MXF SDK

EBUCore Parser

MUX

Essence Processor

Schema-basedEBUCore

object model

Structural Metadata

Essence

EBUCore Document

KLV-basedEBUCore

object model

<<Other metadata parsers>>

OtherKLV-basedmetadata

object models

EBUCoreXML Schema

XSD Code Generator

EBUCoreKLV Dictionary

gen_classesCode Generator

Bi-directional mapping

EBU MXF SDK – Demo

MXF Dump Report

MXF SDK

raw2bmxPCM Audioa0.raw

MPEG-2 LGvideo.raw

ITM504483_EBUcore.

XML

PCM Audioa1.raw

PCM Audioa2.raw

PCM Audioa3.raw

EBU CoreKLV Dictionary

mxflib

mxfdump

Standard MXF KLV Dictionary

playout

vlc

Contents

• Metadata exchanges?• Material Exchange Format• Metadata in MXF• Putting EBUCore into MXF• EBU MXF SDK • EBU MXF SDK – Demo• EBU MXF SDK – Improvements & Community

EBU MXF SDK – Improvements

• Input from the metadata community is welcomed and encouraged

• Experiment, use, and improve the existing code– Instructions on how to use, compile and embed in

the coming weeks

MXF SDK – Metadata Improvements

• Implementation of 377-1 revision changes– Extension of Descriptive Metadata elements with

grouping elements to aid identification of DM elements

• Foreign objects can be identified, but what about their semantics??– Can we store the dictionary in the MXF container?• Solution in proposed

SMPTE 377-2 – KLV Encoded Extension Syntax

Conclusions

• In B2B exchanges of AV material, embedded metadata makes sense

• In MXF files custom descriptive metadata, modeled as a graph of metadata set classes, can be embedded

• As an example, we illustrated how we can map EBUCore to a KLV-encoded representation

• This reference implementation SDK, along with other MXF manipulation tools, is being released as open source software by EBU

Questions? Contact Us!

• http://www.limecraft.com• http://twitter.com/limecraft

• dieter.vanrijsselbergen@limecraft.com