three-dimensional video

ThreeThree--dimensional video:dimensional video: Trends and challengesTrends and challenges

Marco Cagnazzo

Maître de conférences

Télécom-ParisTech

Télécom-ParisTech

Founded in 1878 as Ecole Supérieure de Télégraphie

The place where the word Telecommunications (Télécommunications) was born

Ecole Nationale Supérieure des Télécommunications from 1943 to 2008, Télécom-ParisTech since then

• Member of Institut Mines-Telecom since March 2012

26/02/2014 Marco Cagnazzo 3D Video: Trends and Challenges 2

Télécom-ParisTech

7 Masters Of Science in Telecommunications

Active research

• More than 220 researchers

• ≈400 PhD Students

• 50 Doctorates awarded per year

• Dozens of post-doc positions opened every year

• Over 600 scientific publications per year

CNRS Mixes Research Unity

• Signal and Image Processing

• Computer Science and Networks

• Electronics and Communications


Summary

Introduction

3D scene acquisition and formats

3D geometry

3D representation: coding

3D services

Conclusions


Summary

Introduction • 3D representation: an old new story?

• Depth perception


3D geometry


3D services

Conclusions


3D imaging: an old new story?

Masaccio, Trinità (1425-1427),

Santa Maria Novella, Firenze

Piero della Francesca?, Leon Battista Alberti?, Città ideale (1470-1475 circa),

Galleria Nazionale delle Marche, Urbino.

P. Picasso, Les Demoiselles

d'Avignon, 1907, MOMA, NYC

The weighing of the heart scene from

the Papyrus of Ani, ca. 1200 B.C.

Flat images

Perspective

Perspective, distance fog Multi-view Cubism was based on the idea of

incorporating multiple points of view in a

painted image, as if to simulate the visual

experience of being physically in the

presence of the subject, and seeing it from

different angles (Wikipedia)


Stereoscopic imaging

As soon as photography was born, stereoscopic devices were created

1844: Stereoscope by David Brewster, a device that could take photographic pictures in 3D.

1851: Improvement by Louis Jules Duboscq (picture of Queen Victoria displayed at The Great Exhibition)


Example

Stereoscopic view of Manhattan, 1909


Anaglyph image


3D Movies

1855: the Kinematoscope was invented, ie the Stereo Animation Camera.

1915: The first anaglyph movie was produced in

1922: the first public 3D movie was displayed - The Power of Love

1935: the first 3D color movie was produced

1947: Soviet Union developed 3D films: Robinson Crusoe

’50: many 3D movies were produced: Bwana Devil, House of Wax, Alfred Hitchcock’s Dial M for Murder (movie was released in 2D because not all cinemas were able to display 3D films).

2000s: Computer graphics and 3D Renaissance (Avatar, etc.)

• 3D video channels, 3D TV

• 3D video standards

• Multi-view, super-multiview, holoscopy… holography?


3D Television

2008: 3D broadcast on Japanese cable channel BS 11

01/01/2010: SKY 3D started broadcasting in S. Korea

24/03/2010: Cablevision (USA) launched a 3D version of its MSG channel

03/04/2010: British Sky Broadcasting launched a limited 3D TV broadcast service.

18/05/2010: Spanish Canal+ started 3D broadcast

28/09/2010: Virgin Media launched a 3D TV on Demand service

...

November 2010: 8 3D channels in Europe

April 2011: HIGH TV, a 3D family entertainment Channel launched

2012: 3D TV launched in China, Italy, and other countries

2013: New 3D programs in Brazil; BBC suspends 3D programs


3D Television

ChannelChannel Country(s)Country(s)

HIGH TV 3D Worldwide

n3D United States

Cinema 3D United States

3net United States

Eurosport 3D Europe

Sky 3D United Kingdom

and Ireland

Foxtel 3D Australia

HD1

Belgium (and

other European

countries)

Sky 3D Germany and

Austria

Anixe 3D

Germain-

speaking

countries

3D-TV Finland

Sport 5 3D Israel

MSG 3D United States

nShow 3D Poland

ChannelChannel Country(s)Country(s)

Canal+ 3D France

Canal+ 3D

España Spain

NEXT Man 3D Poland

NEXT Lejdis 3D Poland

NEXT Young 3D Poland

Active 3D India

BS11 Japan

RedeTV! Brazil

Viasat 3D Sweden

Brava3D Europe

Teledünya 3D Turkey

Sky 3D South Korea

Sukachan 3D169 Japan

ESPN 3D United States

Xfinity 3D United States

Penthouse 3D Europe

TV Azteca 3D Mexico


Depth perception

Monocular cues

• Perspective

• Motion parallex

• Depth from motion

• Distance fog and texture degradation

• Object sizes

• Illumination and shades

• Blur

• Occlusions


Monocular cues

Perspective, distance fog and texture degradation


Monocular cues

Depth from motion


Monocular cues

Illumination and shadows


Monocular cues

Defocus blur, occlusions


Binocular cues

Stereovision: vergence

• Disparity perception

Accommodation (focus)


3D Video

Nowadays, 3D video is much about a very simple representation of a 3D scene, i.e., a stereoscopic (two views) representation

However, more complete and flexible representations exist, as we will see

Ideally, one would like to reproduce the light field of the original scene


3D Video Systems

3D

Co

nte

nt

Pro

du

ctio

n

3D

Vid

eo

Co

din

g

3D

Vid

eo

Dec

od

er

+DIB

R

DV

B

Dec

od

er

… …

…

Stereo

camera

Depth

camera

Multi-camera

setup

2D/3D

conversion Video Depth /

Geometry

Meta data

Multi user

3D TV

Single user

3D TV

2D TV

Multiview +

Depth (MVD)


Video services evolution # v

iew

s

# pixels

TV HDTV UDTV

3DTV HD-

3DTV

UD-

3DTV

FTV HD-FTV UD-FTV

720

720

×

576

1920

1080

1920

×

1080

7680

4320

7680

×

4320

2

views

N

views


Summary

Introduction


• Plenoptic function

• Stereo, Multiview, MVD, LDV, holoscopy

3D geometry


3D services

Conclusions


3D Video capture

Stereoscopy : 2 cameras mounted side by side, separated by the same distance as between a person's pupils.

Multi-view capture uses arrays of many cameras to capture a 3D scene through multiple independent videos

Color+depth camera: capture normal video and a depth map, estimated with radar-like techniques (using infrared) or structured light

Multiview+depth (MVD): N Color+depth cameras

• MVD: the most flexible format (view synthesis at user side)

Holoscopy


3D Video representation: plenoptic function

The plenoptic function, or light-field of a scene is the complete information about what can be seen from any angle, at any position, in any time, at any frequence (color)

x

y

z


x

y

z




3D video representation


From the plenoptic function to the stereo video

x

y

z


From the plenoptic function to the multiview video

x

y

z


From the plenoptic function to the super multiview video

x

y

z


3D Video Acquisition: stereo camera


3D Video Acquisition: color + depth


3D Video Acquisition: MVD


3D rendering: anaglyph


3D rendering: polarized glasses


3D rendering: Alternate-frame sequencing

Every second frame is from the left [right] view

Video is projected at twice the frame rate

Viewers wears glasses that shutter alternatively the left or the right eye


Auto stereoscopic display


Traditional 3D rendering: problems

Accommodations (focus) - vergence (disparity) conflict

Cross-talk


From the plenoptic function to the holoscopy

x

y

z


From the plenoptic function to the holoscopic format

New format: holoscopy, or integral imaging

Glasses-free 3D, promising no visual pain

Holoscopic image and videos

Data redundancy

Grid-shaped pattern

Compression?

3D scene Micro-lenses array

Micro-lenses array

Camera 2D screen 3D rendering


Holoscopy


Other formats: Layered Depth Video and Images


3D scene representation: summary

# depths (geometrical information)

# views

LDV

0 depths

1 depths

∞ depths

1 view ∞ views

2D TV Multiview Super Multiview

Holoscopy

Light

field

1View+

Multi

Depth

3D model

+ texture

1View+1

Depth


Summary

Introduction


3D geometry • Pin-hole camera model

• Stereoscopy and disparity


3D services

Conclusions


Pin-hole camera model

C : optical center

f : focal length

c : principal point

Using the image plane we avoid the image inversion of the retinal plane


Pin-hole camera model

Coordinate systems: • W.r.t. the optical center (XC,YC,ZC) • Wr.t. the image plane (x,y) • Wr.t. the principal point (xc,yc) • Real world (X,Y,Z)


Pin-hole camera model M

M’

m’

m

C

f

Zc

m

m’

Image

plane

M

M’

Object

plane


Homogeneous coordinates


Intrinsic parameters

m

m’

Image

plane


Image coordinates


Extrinsic parameters


Image and real coordinates


Stereoscopy

The two projections of the

same point into the two image

planes are called homologous

points

The stereo matching

problem consists in finding

the correspondence

between homologous

points


Epipolar geometry


Parallel cameras

It is a case of particular interest Corresponds to the human vision (frontal vision) Parallel optical axes and same focal length In this case the epipolar lines are parallel to the baseline, and the images

are co-planar Homologous point only differ for the an horizontal component: it is called

disparity It is possible to rectify a couple of camerals, i.e. to produce the images

corresponding to the co-planar case


Disparity and depth

M

B

X

m

X-B

x x'

f

Z

Cl Cr M’


Disparity estimation


The disparity field


The disparity field: example


The disparity estimation problem


Difficulties of stereo matching

Occlusions: not all the left image points are visible in the right image

Not perfectly identical cameras and noise make homologous point having different luminance/colour

Untextured regions: this makes difficult evaluating the data attachment term

Complexity of the minimization problem

• Full search

• Convex minimization

• Parallel algorithms


Post-processing

Often the disparity field can be enhanced using post-processing

• Cross-checking helps in finding occlusion points

• Interpolation: it allows to “fill in” occlusions

• Median filtering: removes estimated values too different with respect to the neighborhood


Summary

Introduction


3D geometry


• Multiview video coding

• MVD video coding

• Holoscopy coding

3D services

Conclusions


Coding of 3D video

Encode separately each view

(Simulcast)

Encode jointly view

• Use other views to perform prediction of current image

Encode one/more views and a depth maps

• Joint or separate coding of view and depth


Compression standards

Frame compatible stereo interleaving

MPEG-2 Multiview Profile

B B B P I B B B I

B B B B P B B B B

P

P


Compression standards: H.264/MVC

H.264 MVC extension

Base view + dependent views

Disparity compensated prediction

BBB BB BBBB B

BB BB BP B PBB

BB BBBB BBBB

BBB BP PBBBB

BBB BB BBBB B

BB BB BP B PBB

BBB BP PBBBB

BBB BP PBBBB

BBB BP PBBBB

BBB BP PBBBB

BBB BP PBBBB

BBB BP PBBBB

BBB BP PBBBB

BBB BI B B B B I

BB BBBB BBBB

0 4 2 042414 4

0 3 1 3 0 3 1 3 0 3

0 2 4 04241 4

3 1 3 0 3 1 3 0 30

0 4 2 042414 4

0 3 1 3 0 3 1 3 0 3

3 1 3 0 3 1 3 0 30

3 1 3 0 3 1 3 0 30

3 1 3 0 3 1 3 0 30

3 1 3 0 3 1 3 0 30

3 1 3 0 3 1 3 0 30

3 1 3 0 3 1 3 0 30

3 1 3 0 3 1 3 0 30

3 10 3 0 3 1 3 0 3

0 2 4 04241 4


3D video coding

3D Video Coding (3DVC)

New phase of standardization in MPEG

Objectives:

• Display-independent representation

• Advanced stereoscopic display processing: e.g. adjust depth perception by controlling baseline distance

• High quality auto-stereoscopic multiview displays: many views with limited bit rate


MVV vs. MVD


3D Video Coding (3DVC)


3D-HEVC: Coding structure

Coding by access units

Temporal

Inter-component

Inter-view (texture)

Inter-view (depth)

HEVC

HEVC + additional tools


Standardization is on-going

Inter-view tools

• Disparity compensated prediction

• Inter-view motion prediction

• …

Inter-component tools

• Quad-tree initialization/limitation

• Motion parameter inheritance

• Intra-prediction inheritance

• …


Enhancing the use of DCP

Temporal Skip Temporal Inter Interview Inter Interview Skip

Intra

DV : 9%

MV : 91%


Conditional mode inheritance


Criteria for inheritance

Sobel

Module

10 20 30 40 50 60

10

20

30

40

50

60

50

100

150

200

250

300

Module

10 20 30 40 50 60

10

20

30

40

50

60

0

50

100

150

200

250

300

350

-2 -1.5 -1 -0.5 0 0.5 1 1.5 20

10

20

30

40

50

60Angle histogram

Angle

Nbr

occure

nces

-2 -1.5 -1 -0.5 0 0.5 1 1.5 20

50

100

150

200

250

300

350Angle histogram

Angle

Nbr

occure

nces


Non standard approach: Depth Coding Based on Elastic Deformations

Base tool: A tool that can find an intermediate contour between an initial and a final one, by generating the geodesic (series of elastic deformations) between the two curves.

2

3

1

4

1

2

3

4

5

6

7

8


Depth compression: impact on image synthesis


DIBR: Depth-image based rendering

Given a view, how to synthesize a virtual view point?

It is possible if depth is known:

Linear operation (omography) in homogenous coordinates

Further simpliflied in the rectified case: disparity compensation

VSRS: view synthesis reference software

mm

CC22 CC11

m'm'

MM

Reference image plane Virtual image plane


VSRS: global scheme

Single view

processing

Single view

processing

Merging

Filling

holes

Reference

homography

matrix

Reference

homography

matrix

Synthesis

homography

matrix


VSRS: single view processing

Depth Map

Synhtesis

View

Synhtesis

Homography

Matrix

Reference

homography

matrix

Synthesis

homography

matrix Reference depth

Reference view

Synthesized view


Depth map synthesis

Mapping of depth values on the image plane

When tow points are associated to the same coordinates,only the nearest is kept (occlusion)

Some coordinates may have no depth value (disocclusion)

Median filtering removes “small” holes

Synthetized depth Median filtering


View synthesis

Mapping of texture values of the reference image using the synthetized depth

Depth knowledge allows to solve some occlusion conflict

Synthesis from the right Synthesis from the left


Contour correction

False contours may appear in the synthetized view

This can me mitigated if filled regions are artificially increased by one pixel


View merging

Left and right images are merged, averaging pixels where both views are available

As a consequence, only small holes remain in the merged image


Holes filling: inpainting


Holes filling: inpainting

Holes

Filling


Encoding holoscopic video

The holoscopic videos (HV) have a lot of redundancy…

… but also a large high-frequency content (grid)

• Grid removal?

Benchmark: “2D coding”, i.e. plain HEVC on the HV


Encoding holoscopic video

Ad hoc techniques

Self-similarities: intra-image motion-estimation

View extraction + Multi-view coding

Scalable coding

Vie

w e

xtr

action

2D Encoder

Residual

encoder

Holoscopic

Prediction

Inter-view

Prediction

Residual

encoder

Multip

lexer


Summary

Introduction


3D geometry


3D services

• FTV and IMVS

Conclusions


FTV System

Video

capture

Decoding View

generation

Pre-

processing Encoding

2D/3D

Display


FTV Display

View

Synhtesis

View

Synhtesis

3D Display

Viewpoint

control

FTV Data

2D/3D

Display


FTV interactive streaming

FTV can be very heavy, even after compression

In the interactive framework, only 2views + 2 depths could be sent

The current view is synthesized using encoded views

Problem: view switching (among encoded views) affects temporal prediction


FTV interactive streaming

Multiview video for free viewpoint TV services

Several view available: the user interactively switches from one view to another

View pattern unknown at encoding time


Interactive Multiview Video Streaming

Views

Time

All frames are Intra

Coded

Each image is coded

and stored only once

Large bandwidth

requested

Relatively low server

space requested



Views

Time

P-frames are used:

all possible frame

dependencies are

coded

Each image is coded

many times

Smallest bandwidth

requested

Very large server

space requested


Distributed video coding: principle

Slepian-Wolf Coder

Quantizer Turbo

Encoder

Min Distort

Reconstr

Q Q’ Buffer

Turbo

Decoder

WZ WZ WZ SI

Image

Interpolation

KF KF Intra

Coder

Intra

Decoder Decoded

KFs

Decoded

WZFs

Encoder Decoder



Views

Time

WZ-frames are used: only parity bits are coded Each image is coded and stored only once Trade-off between server space and bandwidth


Application to IMVS: Interactive Multiview Video Streaming

Bandwidth

Server space

Only

Intra

Predictive coding:

Each image coded

many times

Ideal Case: Path known

at encoding time

WZ coding

Operation

region


Conclusions

3D video has periodically experienced waves of excitement and deception

A main problem is the visual discomfort related to the stereoscopic representation

The emerging format may solve this problem

• Super-multiview, holoscopy

Many problems yet to be solved

• Effective compression

• Quality evaluation (objective and subjective metrics)

• Transmission

Is holography the future?


Conclusions

Contact :

[email protected]

Bibliography : [1] M. Tanimoto, Overview of free viewpoint television. In Signal Processing: Image Communication Volume 21, Issue 6, July 2006, Pages 454-461

[2] A. Smolic and P. Kauff, Interactive 3-D video representation and coding technologies. Proc. IEEE, 93(1), pp. 98–110, Jan. 2005

[3] G. Cheung, A. Ortega and N. Cheung, Interactive Streaming of Stored Multiview Video Using Redundant Frame Structures, in IEEE Transactions on Image Processing, 20(3), pp.744-761, March 2011

[4] F. Dufaux, B. Pesquet-Popescu, M Cagnazzo (eds.): Emerging Technologies for 3D Video. Wiley, 2013

[5] Faugeras, O. , Three-dimensional computer vision: a geometric viewpoint. MIT Press, Cambridge, MA, 1994

[6] C. Fehn, Depth-Image-Based Rendering (DIBR), Compression and Transmission for a New Approach on 3D-TV, SPIE Electronic imaging 2004

[7] M. Bertalmio, G. Sapiro, C. Ballester and V. Caselles, Image inpainting, Computer Graphics, SIGGRAPH 2000, July 2000, 417–424


mailto:[email protected]



THANKS FOR YOUR ATTENTION!

?? || \\ (1)

______________ (1)Questions or comments, ® Dario Rossi, Télécom-ParisTech


three-dimensional video

Technology

d representation

d canal

d geometry

d programs

d channels

d films

d movies

d imaging