Error Concealment

Multimedia Systems and StandardsS2 IF ITTelkom

2

BackgroundEvery communication system has to deal with the problems that may arise during transmission, such as adulteration (bit insertion, deletion or inversion) or loss of the transmitted signalTo solve this problem, a set of techniques were developed, whose purpose was to minimize the influence of the transmission errors at the decoder, taking in consideration the characteristics of the video signal.These techniques are called Error Concealment Techniques

3

Definitionerror concealment is a technique in which an error in a transmitted or encoded signal is replaced by synthetic content, often interpolated from other parts of the signal, in an attempt to produce a more pleasant experience for the viewer or listener, at the cost of concealing the presence of the error in the original content.Error concealment by post-processing at the decoder is one of the important strategies for combating transmission errors in video communication

4

Error Concealment Techniquesthe video decoder attempts to benefit from previously received error-free video information the approximate recovery of lost or erroneous data without relying on additional information from the encoder

5

TechniquesTo estimate and sometimes predict missing video information :

Spatial and temporal interpolation, filtering and smoothingMotion vectors, transform coefficients and administrative bits

6

Techniques (cont’d)take advantage of the human eyes tolerance (high-frequency components than low-frequency components)multi-layer video coding: base & enhancement layersmake use of the spatial and/or temporal correlations between damaged MBs and their neighbouring MBs

7

The goalTo reduce the visual artefacts in segments of a video stream that lie between two error-free synch words. ?: where to insert synch words

Error ConcealmentBasic Idea

Decoder should generate a representation for lost areaMatch as close as possible to the lost infoWithin manageable complexity

TechniquesSpatial Error ConcealmentTemporal Error ConcealmentHybrid ConcealmentOther Techniques 8

9

Spatial Error ConcealmentEmploy the correctly received data of the current frame to conceal the erroneous dataSpatial error concealment methods can be categorized in four main types: interpolation based, stochastic-based, tensor voting-based and exemplar-based.

10

Spatial Error Concealment (Interpolation Based)

conceal each corrupted pixel by interpolation; using the correctly received neighboring pixelsfor instance, directional interpolation methods are proposed that utilize edge directions for interpolationThe weakness of interpolation-based methods is that although they perform very fast, texture preservation is not their intrinsic characteristic and should be imposed separatelyExample : Multi Directional Interpolation

11

Multi Directional InterpolationMDI is one method used to repair the damaged image or video by covering the damaged block using spatial interpolationThe process of MDI are :a. Block Classification and

Directional Decisionb. Block Concealment

12

Multi Directional Interpolation Algorithm

The missing 16×16 MB is recovered 4×4 block by 4×4 block in the proposed error concealment approach.The order of 16 blocks recovered in an MB is from the blocks with dependable available neighbours to the blocks without dependable available neighboursThere are mainly two steps to recover each 4×4 block.In the first step, the block is determined if it is a flat block or a block with one of the eight directional edges according to the estimation of surrounding available pixelsIn the second step, the flat block is recovered using weighted pixel averaging and the block with edge is recovered using Intra_4×4 prediction

13

Multi Directional Interpolation Algorithm

14

Block Classification and Directional Decision

The absence or presence of edges and the most likely direction of the edge, if present, of each 4×4 block in the missing MB are obtained by the use of gradient measures to three layers of pixels of the available adjacent MB as shown in Fig. 1.The local edge gradient components of the pixel x(i,j) in the second layer of the available adjacent MB is computed by:

15

Block Classification and Directional Decision Cont

Which is equivalent to applying 3x3 sobel operator

The angular of gradient at coordinate (i,j) :

And the angle of θ is rounded to the nearest 22.5 degree.

16

Block Classification and Directional Decision Cont

In order to determine if a 4x4 block is lat or with some direction of edge, eight counters Ck, k = 1,....7 are used. The initial value of Ck is set zero.For every pixel x(i,j) and all pixel (y(u,v) to represent any one of these pixels) in the block to be recovered,if y(u,v) satisfy :

Then Ck plus 1/D, where D is given by :

17

Block Classification and Directional Decision Cont

If the largest counter value is below a certain threshold, there is no discernible edge and the block is classified to a flat block

18

Block ConcealmentEach 4×4 block with edge in the missing MB is recovered by Intra_4×4 prediction with edge according to directional decision in the first step.Each flat block is recovered by using the method of weighted pixel averaging.The number of each block in the missing MB is shown

19

Block Concealment ContSince the order of 16 blocks recovered in a MB is from blocks with dependable available neighbours to those without dependable available neighbours, the order of 16 blocks to be recovered will be changed according to the scenario of available neighbour MBsFor examples, if a missing MB has four available edge neighbour MBs, the order of 16 blocks to be recovered is 0, 5, 1, 4, 10, 15, 11, 14, 2, 7, 3, 6, 8, 13, 9, 12;if a missing MB only has up and right available edge neighbour MBs, the order of 16 blocks to be recovered is 5, 4, 1, 0, 7, 6, 3, 2, 13, 12, 9, 8, 15, 14, 11, 10.Furthermore, for each block from 0 to 15, the 16 pixels a, b, …, n, o, p should also be recovered by Intra_4×4 prediction from more reliable pixels X, A, B, C, …, G, H shown at fig 4.

20

Spatial Error Concealment (Stochastic Based)

consider each pixel of the frame as a random variableemploy the Bayes rule to approximate the best pixel values of the corrupted blockThe main disadvantage of these methods is their stationary assumption for input images.

21

Spatial Error Concealment (Tensor Voting Based)

a tensor vector (containing the intensity and gradient information of each pixel) is calculated for each correctly decoded pixelUsing a voting scheme, tensor vectors of the corrupted pixel are approximated and translated to the intensity valuesThe main drawback of tensor voting approaches is their high computational complexity that is not affordable in real-time applications

22

Spatial Error Concealment (Exemplar Based)performed by cloning the best patches from the correctly received region of the image to the corrupted block.In this method, the similarity is measured by the sum-squared difference of the outer boundariesIn this approach, the subblocks near the edge are concealed first.

23

Temporal Error Concealment

Exploits temporal redudancy in a sequence to recover the impaired MacroBlocks (MBs) by utilizing reference frames.Example : Motion Conpensated Temporal Prediction

Temporal Error Concealment

Rely on the continuity of a video sequence in timeUse temporally neighboring areas to conceal lost regionsPrevious Frame Concealment (PFC)

Use previous corresponding data to copy to current frameOnly good when little motionWidely used due to simplicity

24

Hybrid ConcealmentWhen only apply spatial concealment

Concealed regions are significantly blurred

When only use temporal error concealment

Significantly discontinuities in the concealed regions

Hybrid temporal-spatial technique applied

MB mode info of reliable and concealed neighbors decide which concealment method to use

25

Hybrid (cont.)For intra-coded images

Only use spatial concealmentFor inter-coded images

Use temporal concealment when more than half of the available neighbor MBs are inter-codedOtherwise, use spatial concealment

Referred to as Adaptive temporal and spatial Error Concealment (AEC)

26

27

Error DetectionTo activate error concealment techniqueErrors:

loss of synchronization due to error-corrupted VLC parametersthe number of AC coefficients of any 8x8 block of pixels is found to have exceeded 63 The decoded MV component or quantization parameter is outside the acceptable rangetransmission errors

28

Recovery of lost MVs and MB coding modes = INTER

Simplest: replace the erroneous MB by the spatially coinciding MB in the previous frameAlternative: replaced by the motion-compensated MB (what if MV also corrupted?)MV recovery

29

Example: H.263; BER=0.01 %

(a) no concealment, (b) zero-MV technique, (c) MV of spatially corresponding MB in previous frame, (d) MV of MB in previous frame that best moves in the direction of the lost MV

30

Recovery of lost coefficientscan be interpolated from spatially corresponding coefficients: four neighbours

some coefficients damaged use coefficients in the same blockall coefficients are lost using:• four one-pixel wide boundaries• the nearest two one-pixel wide

boundariesmore suitable for INTRA coded blocks

31

Four-one pixel wide boundaries

Based on assumption of continuity of natural scene content in spaceUse pixel values of surrounding available MBs Estimate of lost pixel:

32

αβγ are weighing factorsDetermine relative impact of vertical, Horizontal, upper, lower…

DisadvantageBlurred reconstruction

Four-one pixel wide boundaries

33

the nearest two one-pixel wide boundaries

Selected ResultsPerformance of different error concealment strategies

34