technical university of berlin communication systems group director: prof. thomas sikora carsten...

Technical University of BerlinCommunication Systems Group

Director: Prof. Thomas Sikora

Carsten ClemensClemens@nue.tu-berlin.de

Error Concealment for Stereoscopic Sequences

ITG Fachausschusstagung 3.2, Juni 2006

Error Concealment for Stereoscopic Sequences 2Communication Systems GroupC. Clemens

Technische Universität Berlin

• Monoscopic Error Concealment strategies are not well suited for stereoscopic scenario

• Assumtions

– independently coded views of a stereo image pair

• remaining redundancies between the channels, which can be utilized for error concealment

– block based coding (16x16 blocks)

Introduction

Error Concealment for Stereoscopic Sequences 3Communication Systems GroupC. Clemens

Technische Universität Berlin

Algorithm Overview

Identification of corresponding region– feature extraction– feature matching along epipolar line– selection of matches (M-estimator/RANSAC)

Projective Transformation– initial parameter set from matches– optimization by Newton Method

Smoothing– only in case of discontinuities of depth

Error Concealment for Stereoscopic Sequences 4Communication Systems GroupC. Clemens

Technische Universität Berlin

Matching and Transformation

projective transformation:

Error Concealment for Stereoscopic Sequences 5Communication Systems GroupC. Clemens

Technische Universität Berlin

Selection of feature matches

• M-estimator– uses all matches with different weights– In some cases the transformation fails, because pixels from

outside the image were warped into the erroneous burst.

• RANSAC (random sample consensus)– uses a number of subsamples (four feature matches) – minimize the sum of squared residues of the boundary region:

RANSAC yields better results than M-estimator

Error Concealment for Stereoscopic Sequences 6Communication Systems GroupC. Clemens

Technische Universität Berlin

Adapted Newton Method

• Find the optimal transformation parameter by minimizing a cost function C(k):

• b is the Border Region of the erroneous block burst

T (k ;(xl, yl))

left view right view

b

Error Concealment for Stereoscopic Sequences 7Communication Systems GroupC. Clemens

Technische Universität Berlin

Adapted Newton Method

• Iteration step:

• Problem I: Local minimum solution

• Initial Parameter set is of prime importance

Cost function C(k) over horizontal and vertical translation parameter

Error Concealment for Stereoscopic Sequences 8Communication Systems GroupC. Clemens

Technische Universität Berlin

Adapted Newton Method

• Problem II: Convergence of Newton method

• Successivly decreasing of border pixel size L after every minimum search

Speed of convergence of the Newthon algorithm for different border sizes L

Error Concealment for Stereoscopic Sequences 9Communication Systems GroupC. Clemens

Technische Universität Berlin

3D Block Smoothing

• In case of great discontinuities in depth (variation of disparity) we perform a linear smoothing algorithm towards the surrounding pixel region (3D-BS)

• Minimization of the intersample variance between neighboring samples and to the block borders

Error Concealment for Stereoscopic Sequences 10Communication Systems GroupC. Clemens

Technische Universität Berlin

HQ EC: Results / Example

Error Concealment for Stereoscopic Sequences 11Communication Systems GroupC. Clemens

Technische Universität Berlin

Results / Subjective Evaluation

• Subjective Simulation Results:

– Double Stimulus Continuous Quality Scale Method (DSCQS) as phsychovisual test with 15 subjects

– Shutterglasses (StereoGraphics)

DSCQS method

-0,5

0,5

1,5

2,5

3,5

4,5

5,5

6,5

hall castle tower over all

monoscopic NEWTburst NEWT 3D-BS

DMOS

Error Concealment for Stereoscopic Sequences 12Communication Systems GroupC. Clemens

Technische Universität Berlin

Fast EC: Algorithm Overview

• Block Search– Directional Diamond Search– SAD

• Side Match Distortion ||1 out

jinjsm bb

Nd

Error Concealment for Stereoscopic Sequences 13Communication Systems GroupC. Clemens

Technische Universität Berlin

Fast EC: Matching Example „Hall“

• SMD determined for each possible position• Position with minimum SMD selected• Block used for reconstruction

Error Concealment for Stereoscopic Sequences 14Communication Systems GroupC. Clemens

Technische Universität Berlin

Fast EC: Subjective Evaluation

BALLOON DMOS S.D. conf. interval

Interpolation 5,69 1,19 ±0,65

SBS 4,39 1,12 ±0,62

TBS 2,52 2,64 ±1,45

TSBS 2,49 1,37 ±0,75

Error Concealment for Stereoscopic Sequences 15Communication Systems GroupC. Clemens

Technische Universität Berlin

Simple Matching - Example

• ‘Balloons’ 720x480, corrupted frame and features

Error Concealment for Stereoscopic Sequences 16Communication Systems GroupC. Clemens

Technische Universität Berlin

Simple Matching - Example

• Reference frame (temporal) with features

Error Concealment for Stereoscopic Sequences 17Communication Systems GroupC. Clemens

Technische Universität Berlin

Simple Matching - Example

• Reference blocks

Error Concealment for Stereoscopic Sequences 18Communication Systems GroupC. Clemens

Technische Universität Berlin

Simple Matching - Example

• Reconstructed frame

Error Concealment for Stereoscopic Sequences 19Communication Systems GroupC. Clemens

Technische Universität Berlin

Publications

• K. Günther, C. Clemens, and T. Sikora

A Fast Displacement-Estimation Based Approach For Stereoscopic Error Concealment

PCS 2004, San Francisco

• C. Clemens, M. Kunter, S. Knorr, and T. Sikora:

A hybrid approach for error concealment in stereoscopic images

WIAMIS '04, Lissabon

• M. Kunter, S. Knorr, C. Clemens, and T. Sikora:

A gradient based approach for stereoscopic error concealment

ICIP '04, Singapore

• S. Knorr, C. Clemens, M. Kunter, and T. SikoraRobust Concealment for Erroneous Block Bursts in Stereoscopic Images3D Data Processing, Visualization, and Transmission (3DPVT'04), Thessaloniki, Greece

• K. Günther, C. Clemens, and T. Sikora

A Fast Displacement-Estimation Based Approach For Stereoscopic Error Concealment

PCS 2004, San Francisco

• C. Clemens, M. Kunter, S. Knorr, and T. Sikora:

A hybrid approach for error concealment in stereoscopic images

WIAMIS '04, Lissabon

• M. Kunter, S. Knorr, C. Clemens, and T. Sikora:

A gradient based approach for stereoscopic error concealment

ICIP '04, Singapore

• S. Knorr, C. Clemens, M. Kunter, and T. SikoraRobust Concealment for Erroneous Block Bursts in Stereoscopic Images3D Data Processing, Visualization, and Transmission (3DPVT'04), Thessaloniki, Greece