Overview of Multi-view Video Coding

Yo-Sung Ho; Kwan-Jung Oh;Systems, Signals and Image Processing, 2007 and 6th EURASIP Conference focused on Speech and Image Processing, Multimedia Communications and Services. 14th International Workshop on

Outline

Introduction Applications of Multi-view Video Requirements for Multi-view Video Coding Test Data Sets and Test Conditions Joint Multi-view Video Model (JMVM) Conclusion

Introduction

Introduction

Multimedia Demands holography two-view stereoscopic system with special glasses multi-view video

Multi-view Video: FVV, FVT, 3DTV

What is multi-view video? Why we need multi-view video coding (MVC)? MVC has been studied in the past.

MVP, MCP, DCP, MPEG4 MAC, H.263/4

Application of MVC

Application of MVC

Free Viewpoint Television (FTV) Three-dimensional TV (3DTV) Immersive Teleconference

FTV

What is FTV (Free Viewpoint Television)? Application of FTV:

Entertainment Education Sightseeing Surveillance Archive

FTV

3DTV

What is 3DTV? Interaction may not be required

To broadcast on 3DTV

3DTV

3DTV

Capture by various types of multiple cameras 1D parallel 2D parallel 1D arc …etc.

Intermediate view reconstruction (IVR)

Immersive Teleconference

What is immersive teleconference? Interaction

Requirements for Multi-view Video Coding

Requirements for Multi-view Video Coding

Requirements for multi-view video coding: Compression related requirements System support related requirements

Compression Related Requirements

Compression efficiency View scalability Free viewpoint scalability Spatial/Temporal/SNR scalability Backward compatibility Resource consumption Low delay Robustness

Compression Related Requirements

Resolution, bit depth, chroma sampling format

Picture quality among views Temporal random access View random access Spatial random access Resource management Parallel processing

System Support Related Requirements

Synchronization View generation Non-planar imaging and display systems Camera parameters

Test Data Sets and Test Conditions

Test Data Sets and Test Conditions

Test Data Sets and Test Conditions

Joint Multi-view Video Model

Joint Multi-view Video Model

Random Access

GGOP contains

frames. For accessing any frame within a GGOP, we

have to decode maximum number of frames.

b4(S5/T7), following 18 referencing frames.

Time-first coding order

Encoder Complexity

Minimum decoded picture buffer (DPB)

EX: GOP_length=16, number_of_views=8, the DPB size = 42

MVC codec will have the same coding delay as single view video coding since time-first coding is mandated.

GOP structures and view prediction structure

GOP structures and view prediction structure

GOP structures and view prediction structure

Illumination compensation

ICA MC: illumination change-adaptive motion compensation

Macroblocks(MB) mode in h.264/MPEG-4 AVC: Inter 16*16 mode, Direct 16*16 mode (include B_Skip), and P_Skip mode

DVIC: difference value of illumination change ICA ME: illumination change-adaptive motion

estimation

SAD calculation for the motion estimation of S*T blocks:

In order to compensate the illumination change

1

1 Illumination compensated residual signal

Other Technical Issues View-temporal prediction structure

Single video v.s. Multi-view video Three main coding structures

Encode multiple video sequences separately Utilizes inter-view correlation only Utilizes both temporal and inter-view correlation

View interpolation prediction Decoder side disparity estimation Computing depth at encoder side and transmitting this to t

he decoder Motion/Disparity vector coding

Highly correlated each other

Conclusion

Conclusion

The multi-view video includes multi-viewpoint video sequence captured by several cameras at the same time.

Compress multi-view video efficiently MPEG and JVT are leading the

standardization of MVC.