FTV (Free-viewpoint Television)

Masayuki TanimotoProfessor Emeritus at Nagoya University

Nagoya Industrial Science Research Institute Nagoya, Japan

February 22, 2013

APSIPA Distinguished Lecture 2013APSIPA Distinguished Lecture at Telecom SudParis

APSIPA Mission: To promote broad spectrum of research and education activities in signal and information processing in Asia Pacific

APSIPA Conferences: ASPIPA Annual Summit and Conference 

APSIPA Publications: Transactions on Signal and Information Processing in partnership with Cambridge Journals since 2012; APSIPA Newsletters

APSIPA Social Network: To link members together and to disseminate valuable information more effectively

APSIPA Distinguished Lectures: An APSIPA educational initiative to reach out to the community

Introduction to APSIPA and APSIPA DL

1. Progress of Visual Media2. History of FTV Development3. Principle of FTV4. FTV Technologies and Demo5. Ray-Reproducing FTV6. International Standardization of FTV7. Contributions of FTV8. Conclusions

Outline

1. Progress of Visual Media

Super HDTV demonstrated at the World Expo 2005 in Nagoya(provided by NHK Science & Technical Research Laboratories)

“イ” was displayed on the world’s first electronic TV display developed by Dr. TakayanagiKenjiro in 1926.

Progress of Television in 80 Years

40 scanning lines 4000 scanning lines100 times

To transmit only a partial information (single view) of 3D space

Next Challenge for Visual Media

FTV (Free-viewpoint TV)

To transmit all information (all views) of 3D space

FTV (Free-viewpoint TV)

A visual media that allows users to view a 3D scene by freely changing the viewpoint as if they were there.

TVTV

Progress of Visual Media

More pixels

3DTV3DTV

More views

HDTVHDTV

FTVFTV

More views

UDTVUDTV

More pixels

Num

ber

of

view

s

HD-FTVHD-FTV UD-FTVUD-FTV

HD-3DTVHD-3DTV UD-3DTVUD-3DTV

Number of pixels

Number of Views in Visual Media

⋅⋅⋅⋅⋅⋅⋅

⋅

⋅⋅

FTV

2DTV

Stereoscopic TV

Auto-stereoscopic TV3DTV

1

2

several

infinite

2. History of FTV Development

Process

ing

Capture

Display

Free viewpoint camera

FTV (Free viewpoint TV)

Ray-space

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 year

Free-viewpoint bird’s eye view system

The Matrix

Eye VisionLight field

MPEG-FTV/3DV

MPEG-MVC

MPEG-3DAV

100-camera system

128/48-camera systems

Coding / Interpolation FTV with Audio

FTV on single PC

FTV onPC Cluster

FTV on Mobile Player

[Nagoya University]

[Others]

[MPEG]

Development of FTV Technologies

Pioneering Works on Free Viewpoint

• Multi viewpoint– Movie “Matrix”（1999）

– CMU “Eye Vision” （2001）

• Free viewpoint– Nagoya Univ. “Bird’s Eye View System”（2000）

– Nagoya Univ. “FTV”（2002）

Displayed views are not camera views but generated views.We can set viewpoints anywhere.

FTV

Multi camera

Viewpoint

Captured views

Virtual viewpoint

Generated view

Real cameras

How to realize FTV

Rays

Some views are captured by cameras.Other views are generated by the integration and interpolation of rays.

Free Viewpoint

Makoto Sekitoh, Koichi Toyota, Toshiaki Fujii, Tadahiko Kimoto and Masayuki Tanimoto, “A Virtual Bird's-Eye View System Based on Real Image, ”Electronic Imaging and the Visual Arts 2000 Gifu, 8, pp.8-1‐8-7, October 2000．

Bird’s-Eye View System(The World’s First FTV, 2000)

160×120 pixels, 10 frames/sec

3. Principle of FTV

“ Multiview Imaging and 3TDTV,” IEEE Signal Processing Magazine, Vol.24, No.6, pp.10-21 (November 2007)

12

34

5

X

Y

Z

( , )x y

( , )θ ϕ

Visual Field(Horizontal)

Visual Field(Vertical)

Cameras

Position

Direction

Viewing Zone

2 3 4 5

1

u ( tan )= θ

xy

f x y u( , , )

Visual Field(Horizontal)

Visual Field(Vertical)

Viewing Zone

Ray-Space

45

32

1

Capture of FTV

Real Space

Ray-Space

Horizontalsection

x

u=tanθ

xθtan

y

x

u=tanθ

line detection+ interpolation

Ray-Space Interpolation

Ray-Space and Horizontal Cross-Section

xy

u

f x y u( , , )

Section １

Section ２

Section Image 1

Section Image 2

Ray-Space

Generation of FTV Images

Camera Arrangements for FTV

(b) Circular/spherical arrangementfor convergent view

(a) Linear/planar arrangementfor parallel view

u

v

0°

180°

360°

The horizontal cross-sections of spherical ray-spacehave sinusoidal structures.

Ray-Space in Circular Camera Arrangement

0° 180° 360°

4. FTV Technologies and Demo

Multi-Camera

Captured Images

CorrectedImages

Free-Viewpoint

Images

ReconstructedImages

Ray-Space

Capture Correction Encoder Decoder2D/3DDisplay

Transmission ViewGenerationStorage

M views 1~N views

Configuration of FTV System

4.1 Capture

Capture and Processing Parts of FTV

100-Camera SystemLinear Planar

Circular

4.2 Correction

Before correction After correction

Geometry Correction

(Test sequence of MPEG-FTV : Pantomime)

Color Correction

Geometry: onColor: off

Geometry: onColor: on

4.3 Compression

View（camera）

Time（frame）

Conventional coding：temporal correlationMVC ：temporal and interview correlation

MVC (Multi-view Video Coding)

4.4 View Generation

FTV Demo: AquariumOriginal Camera Views (15 Cameras)

FTV Demo: AquariumGenerated Free Views

Original Views

Generated Views - Interpolated -

Generated Views - Forward and Backward -

FTV on PC(Real-Time FTV on a laptop PC)

4.5 Display/User Interface

FTV on a laptop PC                      FTV on a mobile player   FTV on an all‐aroundwith mouse control                     with touch panel control  3D display  (Seelinder)

FTV on a 2D display with head tracking

FTV on a multi‐view 3D display with head tracking

FTV on a multi‐view 3D display without head tracking

Interface of FTV

FTV on a Mobile Player

2D Display with View Control

Multiview 3D Display

without eye tracking with eye trackingviewing zone: wide crosstalk: eliminated

viewing zone: narrow, periodical crosstalk: large

Super-Multiview 3D Display: The SeeLinder

4.6 FTV for Static Scenes

Captured Video

Generated Free-viewpoint Images

Actual Camera Movement

Virtual Camera Movement

Demo 1 (Indoor Scene)

Generated Free-viewpoint Images

Captured Video

Actual Camera Movement

Virtual Camera Movement

Demo 2 (Outdoor Scene)

5. Ray-Reproducing FTV

Ray-Reproducing FTV

It captures all rays in a real 3D space and reproduces the same rays at different place and time.Thus, it provides us the same viewing as in the real world.

Object

Double

hyperbolic mirror

Galvanometer mirrorHigh-speed

Camera Real image of object

Configuration of Mirror-Scan System

Mirror-Scan Ray Capture (55 degree)

High Frame-rate Camera

Metal Halide Lamp

Parabolic Mirror x 2

Object Lower

Focal Point

Optical Image

of Object

Rotating Mirror

High Frame-rate Camera

Rotating Mirror

(45 deg Slanted)

Motor

Upper

Focal Point

Object

Motor

Acrylic Board

All-Around Ray Capture by Mirror-Scan 1

Captured Views of All-Around Mirror-Scan System

Milk drop into water(Rotating mirror：30 rotations/s, High-speed camera：10,800 fps)

Pig

Rays of 2D Display

Projected objects

Only control of magnitude needed

Display

Real objects

2D display emits rays with equal magnitude in all directions

Objects

rays from display

rays from objectsDisplay pixels

Control of magnitude and direction needed

Ideal 3D display emits rays with different magnitude in each direction

Rays of Ideal 3D Display

move

Principle of Ray Control

1-dimensional light source array

move

Parallax barrier

Cylindrical Parallax Barrier(Fast Revolution:1800rpm)

1-D Light Source Array(LED array)(Slow Revolution:100rpm)

Structure of the SeeLinder

• Directions of the rays are scanned by revolution.

• Intensity of each light ismodulated in synchronization with revolution.

All-Around Ray-Reproducing Display: The SeeLinder

High Speed Camera

Display

Distortion Correction/Data Conversion

Metal Halide Lamp

Mirror-Scan

All-Around Ray-Reproducing FTV

Demo of All-Around FTV

6. International Standardization of FTV

We proposed FTV to MPEG in 2001 and have been promoting its standardization.As the first phase of FTV, MVC (Multi-view Video Coding) started in 2004 and was completed in 2009. MVC has been adopted by Blu-ray 3D.As the second phase of FTV, 3DV (3D Video) started in 2007 and currently in progress at Joint Collaborative Team of MPEG and ITU-T.

FTV Standardization in MPEG

2001/12

Proposal of FTV

2003/10

CfCon 3DAV

CfEon MVC

CfPon MVC

3DAV seminar

2002/12 time

2004/10

2005/07

MVC (moved to JVT in 2006/07)

Evaluation of proposals

2006/01

Start MVC (Multi-view Video Coding)

2007/07

Req. on FTV

3DV (moved to JCT-3V in 2012/07)

2004/03

3DAV

•Requirements on 3DAV•EEs on 3DAV

•CEs on MVC

2007/04Start MPEG-FTV

2001/12Start 3DAV

2009/05MVC completed

Targets converged on FTV

Second phase of FTV

First phase of FTV

2009/02Requirements on 3DV

History of FTV Standardization in MPEG

CfPon 3DV

2011/03

3D video

Omni-directional

FTV

Stereo TV

Stereo TV with depth

Topics Discussed at 3DAV

FTV

Converged on FTV andMVC startedin March 2004

6.1 MVC

First Phase of FTV in MPEG

MVCEncoder

Sender side Receiver side

Trans-mission

MVCDecoder

Standardized in 2009Adopted by Blu-ray 3D

MVC (Multi-view Video Coding)

multi-view

6.2 3DV

Second Phase of FTV in MPEG

Depth Estimation

3DVEncoder

Stereo camera

Sender side Receiver side Multi-viewdisplay

Trans-mission

3DVDecoder

View Generation

multi-view

multi-depth

3DV (3D Video)

MVD (multi-view + depth)

Capture Correction/Conversion Encoder Decoder Display

TransmissionSynthesis

Storage

Multi-Camera Captured Multi-View Images

Corrected Multi-View Images and Multi-View Depth

Free ViewpointImages

Reconstructed Multi-View Images and Multi-View Depth

Test sequences

Depth EstimationReference Software

FTV Reference Model

View SynthesisReference Software

①

②

③

Contributions of Nagoya Univ. to 3DV

FTV Test Sequenceshttp://www.tanimoto.nuee.nagoya-u.ac.jp/

kendo balloons

pantomime champagne_tower dog

Rena Akko & Kayo

Kendo and balloons are captured by moving camera array

Champagne_tower

Balloons

July 5, 200781st MPEG, Lausanne

July 18, 2012101st MPEG, Stockholm

Development of 3DV

Participants: 6Input documents: 3

Participants: 150Input documents: 170

FormatConversion(optional)

Decoder View Synthesis(VSRS or else)

or

Framework of CfP for 3DV Standardization

Encoder

3-view case

2-view case

PSNR Subjective Test

MVD (multi-view + depth)

Alternative 3D Format

MVC+D (encode view and depth independently)

3D-AVC (depth is used to encode view)

3D-HEVC (depth is used to encode view)

Standardization Tracks for Coding

MVD (Multi-View plus Depth)

GVD (Global View and Depth)

Warp

Data Formats for 3D

MV (multi-view)

Data Formats for Television

MVD (multi-view + depth)

GVD(Global View and Depth)

1 view+ 1 depth

1 view 2 views

[2DTV] [Stereoscopic TV] [3DTV]

MVD

Multi view

GVD

Global view

Multi depth

Global depth

left center right

Comparison between MVD and GVD

Comparison Between GVD-NISRI and GVD-NICT

Global view

Global depth

GVD‐NICT

left center right

Global view

Global depth

GVD‐NISRI

7. Contributions of FTV

Multiview has been popular in special sessions and special issues of international conferences and journals since 2007.

FTV has been listed in the fields of interest and session themes of international conferences and journals since 2009.

Creating New Research Fields

18 - 20 January, 2010San Jose Convention Center, San Jose, CA United States

Session 5: DIBR and FTV (Depth Image Based Rendering and Free Viewpoint Television) Session Chair: Janusz Konrad, Boston Univ.

Paper 7524-17Quality improving techniques for free-viewpoint DIBRPaper 7524-18Structured light-based high-accuracy depth imaging applied for DIBR in multiview3DTVPaper 7524-19Novel view synthesis with residual error feedback for FTV

IS&T/SPIE Electronic Imaging 2010 Stereoscopic Displays and Applications XXI

S1 Coding and Processing for Advanced 3DTV/FTV (Special oral)Chair: Masayuki Tanimoto, Nagoya University, Japan

S1-1 3D Video Coding Using Advanced Prediction, Depth Modeling,and Encoder Control MethodsHeiko Schwarz, Christian Bartnik, Sebastian Bosse, Heribert Brust,Tobias Hinz, Haricharan Lakshman, Detlev Marpe, PhilippMerkle,,Karsten Müller, Hunn Rhee, Gerhard Tech, Martin Winken, Thomas WiegandFraunhofer Institute for Telecommunications Heinrich Hertz Institute, Germany

S1-2 Coding of Multiple Video+Depth Using HEVC Technology and Reduced Representations of Side Views and Depth MapsMarek Domański, Tomasz Grajek, Damian Karwowski, Krzysztof Klimaszewski, Jacek Konieczny, Maciej Kurc, Adam Łuczak, Robert Ratajczak, Jakub Siast, Olgierd Stankiewicz, Jakub Stankowski, Krzysztof WegnerPoznań University of Technology, Poland

S1-3 A Framework of 3D Video Coding Using View Synthesis PredictionCheon Lee, Yo-Sung HoGwangju Institute of Science and Technology, South Korea

S1-4 Robust Joint Reconstruction in Compressed Multi-view ImagingQionghai Dai, Changjun Fu, Xiangyang Ji, Yongbing ZhangTsinghua University, China

S1-5 Three-Dimensional Television System Based on Integral PhotographyJun AraiJapan Broadcasting Corporation, Japan

S1-6 3D Visual System Using Ray-Based Image Sensors and Electronic Holography DisplayKenji Yamamoto, Yasuyuki Ichihashi, Takanori Senoh, Ryutaro Oi, Taiichiro KuritaNational Institute of Information and Communications Technology, Japan

Emerging Techniques for Next Generation Video/Image Coding‐ Content‐ and context‐adaptive coding techniques‐ Joint codec optimization w/ context information‐ Parallel‐friendly coding algorithms‐ Perceptual coding and quality assessment‐ Non‐traditional coding techniques‐ Representation and analysis of visual signalsAdvanced Techniques for Emerging 3D Videos‐ Representation, analysis and coding of 3D scenes‐Multi‐view video processing and coding‐ 3D videos and free viewpoint TV‐ 2D to 3D video conversion‐ Graphics representations and content creationVisual Communications‐ Video streaming and networking systems‐ Internet video and QoS issues‐ Video on demand and live video services‐Wireless and broadband videos‐ Secure and robust video transmissions

Systems and Techniques for Human Interaction‐Mobile visual search‐ Futuristic human‐machine interfaces‐Multimodal recognition for mixed reality‐ Natural image to 2D/3D virtual translation‐ Display design for mixed reality‐ Virtually real and mixed reality applications‐ Biomedical imagingEmbedded Systems and Architecture Implementations‐ Concurrent algorithm/architecture optimization‐ Embedded software for DSP, ASIP, VLIW, multi‐core CPU, GPU, etc.‐ Architecture based design on VLSI, FPGA, reconfigurable architectures, SoC, etc.Cloud Multimedia Systems, Applications and Services‐ Infrastructure for media storage/distribution & computing‐ Performance metrics for multimedia processing in clouds‐ Large‐scale multimedia content processing and analysis‐ Complex semantic multimedia computing in clouds‐Mobile applications and services over clouds

Contribution to FIFA World Cup

2022 FIFA World Cup Japan Bid CommitteePlan to deliver the excitement on soccer stadium to the world by FTV

free navigation

realistic viewing

Innovation of FIFA World Cup Viewing by FTV

Free-viewpoint Vision

Viewing by free navigation

Full Court 3D Vision

Realistic 3D viewing

http://www.dream-2022.jp/jp/our_bid/bid_book/

“Revolutionizing Football” Produced by2022 FIFA World Cup Japan Bid Committee

TV Asahi “Hodo Station” Introducing FTV on December 1, 2010

Report on the final situation on FIFA World Cup Voting in ZurichThe key of Japanese plan is application of FTV to the World Cup.

We proposed the concept of FTV and realized it by developing ray capture, processing and display technologies.FTV is the ultimate 3DTV and ranked as the top of visual media.The international standardization of FTV has been conducted in MPEG.FTV enables realistic viewing and free navigation of 3D scenes as planed for FIFA World Cup.

8. Conclusions