diepold - multimedia at your fingertips 1 open standards – the fuel for ubiquitous multimedia...

Diepold - Multimedia at Your Fingertips 1

Open Standards – the Fuel for Ubiquitous Multimedia Deployment

Klaus DiepoldTechnische Universität München

Munich, Germany

Multimedia at Your FingertipsDelft, November 2006


Open Standards

Open Standards vs. Proprietary (Closed) Standards

I like Standards – there are some many to choose from

Standards for Mulitmedia MPEG Standards (ISO/IEC)


Standardisation – the MPEG Way

Standard: The process by which a group of individuals recognise the advantage of all doing certain things in an agreed way and codify that agreement.


Standards Help Markets Mature

Compatibility over device generations

Interoperability between vendors

Minimize dependency on a single vendor

No one company can service all the needs

No one implementation is best for every use

Standards facilitate competition on the product or service level

Benefit for the Consumer


Standards Achieve High Quality

Specification captures best core technologies

Standards provide many degrees of freedom

Law of large numbers will prevail

Cast of thousands working in the open to improve quality

Economy of scale


Standards Survive the Test of Time

NTSC – 1953

PAL – 1963

VHS -1976 (industry standard though)

Audio CD - 1980

TCP/IP - 1981

MPEG-1, MP3 - 1992

MPEG-2 – 1994

DVB – 1994

DVD – 1995


1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

JPEG “Still images”

MPEG-1 “Digital Storage Media”

MPEG-2 “Digital TV and DVD”

MPEG-4 “Interactive Multimedia Objects”

MPEG-7 “Interface for Content Description”

MPEG-21 “Multimedia Framework”

JPEG2000 “Still images”

H.262H.262 “Digital TV and DVD”

H.261H.261 “p64”

H.263H.263 “Low Bitrate Coding (LBC)”

H.264H.264 “Digital TV and DVD”MPEG-4 Part 10 (AVC)

H.263+H.263+ “Very LBC”

ISO/IEC Standards ITU-T Standards

H.120H.120 1984/1988

T.800T.800

T.81T.81

JPEG-LS “Lossless Coding of Still Images” T.87T.87

MPEG-A “Multimedia Application Formats”New !


MPEG-A: Multimedia Application Formats

Large Number of diverse MPEG Standards

difficult to oversee all the technical options

evaluate all their relative benefits.

Collection of tools is no longer enough

Complete solutions are needed

Industry consorita define application specific combinations of standardized tools

DVB, DVD,…

3GPP…

Etc.

New !


Multimedia Application Formats: The Super-Format Challenge

Objective for MAFs

Provide combinations of MPEG tools

Provide a complete multimedia solutions

Satisfy the needs of certain classes of applications

Consider the need for fast standardisation

Application specific super-formats

New !


Multimedia Application Formats: Concept

MP

EG-1

MP

EG-2

MP

EG-4

MP

EG-7

MP

EG-21

JPE

G

MP

EG-A

Photo Player MAF

Music Player MAF

...

Other MAF

ISO

/IEC

11172

ISO

/IEC

13818

ISO

/IEC

14496

ISO

/IEC

21000ISO/IEC 23000

ISO

/IEC

15938

...

ISO

/IEC

10918

JPE

G 2000

ISO

/IEC

15444

MP

EG-1

MP

EG-2

MP

EG-4

MP

EG-7

MP

EG-21

JPE

G

MP

EG-A

Photo Player MAF

Music Player MAF

...

Other MAF

ISO

/IEC

11172

ISO

/IEC

13818

ISO

/IEC

14496

ISO

/IEC

21000ISO/IEC 23000

ISO

/IEC

15938

...

ISO

/IEC

10918

JPE

G 2000

ISO

/IEC

15444


The MAF Story So Far … (ISO/IEC 23000)

1: Purpose of Multimedia Application Formats (TR)

2: Music Player - includes Music Player and Protected Music Player (FCD in October 2006)

3: Photo Player (FDIS in October 2006)

4: Musical Slide Show (FCD in July 2006)

5: Media Streaming (CD in October 2006)

6: Professional Archival (WD in October 2006)

7: Open Release (CD in October 2006)

8: Portable Video Player (WD in October 2006)

9: Digital Multimedia Broadcasting (WD in October 2006)


Music Player MAFMusic Player MAF


Music Player MAF

Digital music libraries

Combination of audio, metadata, and images

Handling audio data, metadata, and images

for individual pieces of music

collections of music

complete albums or playlists.


Music Player Format: Main Tools

MP3 audio coding (MPEG-1/2 Audio Layer III)

MPEG-7 metadata (corresponding to ID3-Tags)

MPEG-21 Digital Item Declaration (DID)

ISO Media Base, MPEG-4 and MPEG-21 File Formats

JPEG images


Creating Music Player Files …

...Express ID3tagsas

MPEG-7 Metadata

FormatMP3 on MP4

MP3 audio file with ID3 tags

JPEGImageData

MP21 File

JPEGImageData

MPEG-7Metadata

MP3AudioData

JPEGImageData

MPEG-7Metadata

MP3AudioData

...

...

...

MPEG-21 DID

...Express ID3tagsas

MPEG-7 Metadata

FormatMP3 on MP4

MP3 audio file with ID3 tags

JPEGImageData

MP21 File

JPEGImageData

MPEG-7Metadata

MP3AudioData

JPEGImageData

MPEG-7Metadata

MP3AudioData

...

...

...

MPEG-21 DID


Playing Music Player Files …


Photo Player MAFPhoto Player MAFFDIS in October 2006FDIS in October 2006


Photo Player MAF

Digital photos as personal media;

Amount of digital photos is difficult to be managed and accessed …

Difficult to search and find

A standard file format for

easy management

organization of digital photos


Photo Player MAF: Main Tools

MPEG ISO Base Media and MPEG-4 File Formats

MPEG-7 Systems (Binary Format for Metadata - BIM)

MPEG-7 Multimedia Description Schema metadata

MPEG-7 Visual metadata

Dominant Color Descriptor, Scalable Color Descriptor

Color Layout Descriptor, Color Structure Descriptor

Edge Histogram Descriptor, Homogeneous Texture Descriptor

JPEG images

EXIF (Exchangeable Image File Format) acquisition metadata


Photo Player MAF Capabilities …

Photo Categorization

Situation-based Photo Clustering


ProtectedProtected Music Player Music Player MAFsMAFs

FCD in October 2006FCD in October 2006


Protected Music Player MAFs

Content protection for Music

Different approaches to protection

different understandings of interoperability.

Different protection approaches

signaled using MPEG-21 Intellectual Property Management and Protection (IPMP) Components tools.


Music Player Format: Main Tools

MP3 audio coding (MPEG-1/2 Audio Layer III)

MPEG-7 metadata (corresponding to ID3-Tags)

MPEG-21 Digital Item Declaration (DID)

ISO Media Base, MPEG-4 and MPEG-21 File Formats

JPEG images

MPEG-21 Rights Expression Language (REL)

MPEG-21 IPMP Components

Outside MPEG: AES128 encryption


Musical Slide Show MAFMusical Slide Show MAFFCD in October 2006FCD in October 2006


Musical Slide Show MAF

Slide shows, karaoke, story telling, photo-music albuns, practice materials of foreign languages, etc.

Based the Music Player MAF and the Photo Player MAFs

+ text and animation

+ synchronization capabilities.


Musical Slide Show Content …

Synchronized text (Lyric 1)




Image 1 Image 2 Image 3 Image 4 Image 5

Image Samples

t

0 sec 5 sec 7 sec 10 sec

0 sec 4 sec 8 sec 12 sec

Images Time stamps

Text Time stamps

14 sec

LASeR Script

animation animation animation animation animation

Text Samples


Musical Slide Show MAF: Main Tools

MPEG-1/2 Layer III (MP3) audio

ISO Base Media and MPEG-4 File Formats

MPEG-4 Streaming Text Format for timed text

MPEG-4 LASeR, Mini profile for animation and composition

Lightweight Animated Scene Representation

MPEG-7 metadata

MPEG-21 DID and File Format

JPEG images


Media Streaming MAFMedia Streaming MAF CD in CD in

October 2006October 2006


Media Streaming MAF

Proposed by the Digital Media Project (DMP) Consortium

Public broadcasting in the digital world

Commercial broadcasting in the digital world

Internet streaming via Multicast

Pay TV

IPTV

Internet Television

Storage and subsequent play of content in a domain


Media Streaming MAF: Main Tools

MPEG-2 and MPEG-4 IPMP-X

MPEG-21 DID and Digital Item Identification (DII)

MPEG-21 IPMP Components

MPEG-21 REL

ISO Base Media and MPEG-21 File Formats

MPEG-21 Binary Format

MPEG-21 Digital Item Streaming (DIS)

Supplemental media technologies, e.g. domain management


Open Release MAFOpen Release MAFCD in October 2006CD in October 2006


Open Release MAF

“Public Domain” Content / Non-Commercial Contene

e.g., collections of photos, teaching lectures, teasers, advertizing material, etc.,

Distribution of content for free – without loosing track of the usage

e.g. automatically sending email back to the file provider …

Express intention of Creative Commons (CC) licenses

No enforcement


Open Release MAF: Main Tools

Subset of MPEG-21 DID and MPEG-21 DII

Subset of MPEG-21 REL

Subset of MPEG-21 Rights Data Dictionary (RDD)

MPEG-21 File Format

MPEG-21 Binary Format

MPEG-21 Event Reporting


Professional Archival MAFProfessional Archival MAFWD in October 2006WD in October 2006


Professional Archical MAF

Digital backup and direct delivery of recorded music

Compression and archiving intermediate data generated by sound editing tools

Preservation & archiving

Packing file format for non-audio waveforms


Professional Archical MAF: Main Tools

MPEG-4 Part-3: Audio Lossless Coding (ALS)

MPEG-4 Part-3: Lossless coding of oversampled audio (DST)

ISO Base Media File Format

MPEG-4 File Format

MPEG-7 MDS metadata

MPEG-21 DID

MPEG-21 DII

MPEG-21 File Format


Portable Video Player MAFPortable Video Player MAFWD in October 2006WD in October 2006


Portable Video Player MAF

Standardised “Video iPod”

Playback of personal recordings or home videos

Playback of content sold on disk media (mini-DVD)

Playback of content offered via the Internet


Portable Video Player MAF: Main Tools

ISO Base Media, MPEG-4

Advanced Video Coding (AVC) Baseline Profile

MPEG-4 HE-AAC Profile, Level 2, Stereo

MPEG-4 BSAC Audio

MPEG-4 Streaming Text Format for subtitles

MPEG-4 LASeR (e.g. for menu screens)

MPEG-7 Metadata

JPEG still images of e.g. movie posters

MPEG-21 File Format and DID

MPEG-21 based Content Protection Mechanisms


Digital Multimedia Digital Multimedia Broadcasting MAFBroadcasting MAF

WD in October 2006WD in October 2006


Digital Multimedia Broadcasting MAF

Storage of mobile broadcast content

IP media services

User-creative mobile broadcasting content

Mobile TV Services in Korea (DMB)


Digital Multimedia Broadcasting MAF: Main Tools

ISO Base Media File Format

Video: Advanced Video Coding (AVC)

Audio: MPEG-1 Layer-2, MPEG-2 AAC+, MPEG-4 ER-BSAC MPEG-4 HE-AAC

Systems: IOD/OD, BIFS, MPEG-4 File format, AVC File format

DMB Data Services

Content description metadata: MPEG-7, TV-Anytime etc.

User metadata: MPEG-7, TV-Anytime, etc

MPEG-21 IPMP, REL, etc.


Surveillance MAF

Follows significant interest on surveillance applications by many companies.

Currently, this MAF proposes to include:

MPEG-2 Video, MPEG-4 Visual, AVC,SVC

MPEG-4 Audio (scalable coding)

MPEG-2/MPEG-4 Systems (media streaming)

MPEG-7 Simple profile

MPEG-7 Visual and Audio

MPEG-21 DID, DII, DIA

MPEG-21 IPMP Components and REL

MPEG-X File Format

Cryptographic functions from outside


Digital Video/Cinema MAF

Addresses digital entertainment, especially digital cinema.

Currently, this MAF proposes to include:

MPEG-2 Video (main profile)

MPEG-4 AVC/SVC

MPEG Audio

MPEG-7 Visual

MPEG-21 File format

MPEG-21 DID/ DII

MPEG-21 IPMP/REL/RDD

MPEG-21 Digital Item Adaptation


Final Remarks

A number of MAFs are ready for application development

ubiquitous multimedia deployment

MPEG-A represents a new layer for interoperability.

MPEG-A provides complete solutions

MPEG-A teaches the usage of MPEG standards

e.g. MPEG-7 and MPEG-21.

However, the use of Standards is challenged by Patent Issues


Useful References

MAFs Overview Document available at

http://www.chiariglione.org/mpeg/

To probe further:

K. Diepold, F. Pereira, W. Chang. MPEG-A: Multimedia Application Formats. IEEE Multimedia, Volume 12, Issue 4, Page(s): 34 – 41, Oct.-Dec. 2005


Thanks for your attention !

Thanks to

Prof. Munchurl Kim and Prof. Fernando Pereira

for some slides.

diepold - multimedia at your fingertips 1 open standards – the fuel for ubiquitous multimedia...

Documents

multimedia frameworkjpeg2000

conceptdiepold multimedia

consumerdiepold multimedia

digital tv

mpeg waystandard

combinations of mpeg

use standards

protected music player