A hybrid error concealment A hybrid error concealment scheme for MPEG-2 video scheme for MPEG-2 video transmission based on best transmission based on best neighborhood matching algorithmneighborhood matching algorithm

Li-Wei Kang and Jin-Jang Leou

Journal of Visual Communication & Image Representation 2005

OutlineOutlineIntroductionProposed hybrid error

concealment scheme for MPEG-2 intra-coded I frames

Proposed hybrid error concealment scheme for MPEG-2 inter-coded (P or B) frames

Simulation results

Introduction (1/2)Introduction (1/2)Transmission errors and error

propagation

Network

Encoded stream

Encoded stream

Original pictures

Decoded pictures

Introduction (2/2)Introduction (2/2)Error control

◦Channel coding

◦Error resilient coding

◦Detection and correction

◦Detection and concealment

Encoded stream

Encoded stream

Encoded stream + = Encoded

stream

currentprevious future current

Encoder control

Decoder control

Proposed hybrid error Proposed hybrid error concealment scheme for concealment scheme for MPEG-2MPEG-2I frame

◦Spatial search for similar blocks◦Selection between H.26L test model

and FBNM P frame

◦Temporal search for similar blocks◦Determination of block behavior

Small motions, middle motions, and large motions

◦Search for motion vectors

Proposed hybrid error Proposed hybrid error concealment scheme for concealment scheme for MPEG-2 intra-coded I frames MPEG-2 intra-coded I frames (1/7)(1/7)H.26L test model [25]

Best neighborhood matching (BNM) [15]

a1

a2

a3

a4

4321 2222),( a

Nja

Nia

BjNa

NiNjif

N+mN+m

H

L

Best match

Proposed hybrid error Proposed hybrid error concealment scheme for concealment scheme for MPEG-2 intra-coded I frames MPEG-2 intra-coded I frames (2/7)(2/7)Proposed fast BNM

◦Step 1: Initial candidate block by MAD

◦Step 2: Hexagon-based fast search by MAD

Proposed hybrid error Proposed hybrid error concealment scheme for concealment scheme for MPEG-2 intra-coded I frames MPEG-2 intra-coded I frames (3/7)(3/7)Proposed hybrid error concealment

scheme◦Fitness function

H.26L or fast BNM? Check the smoothness of the boundaries

between the concealed block and its neighbors Check the similarity between the concealed

block and its neighbors Components

AIDB AMD AVD

Proposed hybrid error Proposed hybrid error concealment scheme for concealment scheme for MPEG-2 intra-coded I frames MPEG-2 intra-coded I frames (4/7)(4/7)Average intersample difference

across the block boundaries between a concealed block and its four neighbors (AIDB)◦Smoothness of the boundaries

1

0

1

0

1

0

1

0

ofneighbor right theis if ,)0,()1,(1

ofneighbor left theis if ,)1,()0,(1

ofneighbor bottom theis if ,),0(),1(1

ofneighbor top theis if ,),1(),0(1

),(

N

i

N

i

N

j

N

j

BXiXNiBN

BXNiXiBN

BXjXjNBN

BXjNXjBN

XBABD

Bi BNB

iB

BBABDBN

BAIDB ),(1)(

Proposed hybrid error Proposed hybrid error concealment scheme for concealment scheme for MPEG-2 intra-coded I frames MPEG-2 intra-coded I frames (5/7)(5/7)Average mean difference

between a concealed block and its eight neighbors (AMD)

Bi BNB

iB

BmeanBmeanBN

BAMD )()(||

1)(

mean

meanmean

mean

meanmeanmean

mean

mean

Proposed hybrid error Proposed hybrid error concealment scheme for concealment scheme for MPEG-2 intra-coded I frames MPEG-2 intra-coded I frames (6/7)(6/7)Average variance difference

between a concealed block and its eight neighbors (AVD)

Bi BNB

iB

BBBN

BAVD )var()var(||

1)(

Proposed hybrid error Proposed hybrid error concealment scheme for concealment scheme for MPEG-2 intra-coded I frames MPEG-2 intra-coded I frames (7/7)(7/7)Fitness function

Choose of concealed block

◦Purpose of TC: BH.26L is usually smaller than BFBNM, but it’s visual quality is not always better than BFBNM

)()()()( BAVDBAMDBAIDBBCF

o.w.

)()( if 26.26. CFBNMLH

FBNM

LHC

TBCFBCFBB

B

Proposed hybrid error Proposed hybrid error concealment scheme for concealment scheme for MPEG-2 inter-coded (P or B) MPEG-2 inter-coded (P or B) frames (1/2)frames (1/2)MV magnitude

Determination of types of the corrupted blocks

Block matching

bN

ibiaveNMVMV

1

/

Small motion blocksCandidates:MV0

MVave

Middle motion blocksCandidates:MVi for i = 1 to 8MVave

MVmedium

Large motion blocksCandidates:Fast search algorithm

Tl Th

Proposed hybrid error Proposed hybrid error concealment scheme for concealment scheme for MPEG-2 inter-coded (P or B) MPEG-2 inter-coded (P or B) frames (2/2)frames (2/2)Fast search algorithm for large

motion blocks

Search range for the corrupted block

2D+1

2D+1

Simulation results (1/8)Simulation results (1/8)MPEG-2

◦GOP size = 12◦IBBP structure◦1.5 Mbps◦30 fps◦One packet contains one slice

CIF◦Coastguard◦Football◦Salesman◦Table tennis

Simulation results (2/8)Simulation results (2/8)parameters

◦I frame N = 16; m = 4; L = H = 120

◦P or B frame Tl = 8; Th = 12; TC = 200; D = 2

N+mN+m

H

L

o.w.

)()( if 26.26. CFBNMLH

FBNM

LHC

TBCFBCFBB

B

Tl Th

Simulation results (3/8)Simulation results (3/8)Discussions

◦H and L for I frame Smaller: good candidates may be excluded Larger: bad candidates may be included

◦Tl and Th for P frame Smaller: better concealment results but high

computation cost Larger: worst concealment results but low

computation cost◦D for P frame

Smaller: worst concealment results but low computation cost

Larger: better concealment results but high computation cost

Simulation results (4/8)Simulation results (4/8)1. replace all corrupted blocks by

zeros2. I frame: H.26L; P/B frame:

overlapped region matching [21]3. I frame: split matching; P/B frame:

forward-backward block matching [22]

4. I frame: H.26L; P/B frame: neighbor matching [23]

5. Proposed

MV interpolation

Simulation results (5/8)Simulation results (5/8)Football

Salesman

Processing time (10% PLR)

12 34

Simulation results (6/8)Simulation results (6/8)Visual quality (15% PLR)

Simulation results (7/8)Simulation results (7/8)Visual quality (15% PLR)

Simulation results (8/8)Simulation results (8/8)FootballPLR = 15%

ReferencesReferences[15] Z. Wang, Y. Yu, D. Zhang, Best neighborhood matching: an

information loss restoration technique for block-based image coding systems, IEEE Trans. Image Process. 7 (7) (1998) 1056–1061.

[21] M.C. Hong, L. Kondi, H. Scwab, A.K. Katsaggelos, Video error concealment techniques, Signal Process.: Image Commun. 14 (1999) 437–492.

[22] S. Tsekeridou, I. Pitas, MPEG-2 error concealment based on block-matching principles, IEEE Trans. Circuits Syst. Video Technol. 10 (4) (2000) 646–658.

[23] S. Valente, C. Dufour, F. Groliere, D. Snook, An efficient error concealment implementation for MPEG-4 video streams, IEEE Trans. Consumer Electron. 47 (3) (2001) 568–578.

[25] Y.K. Wang, M.M. Hannuksela, V. Varsa, A. Hourunranta, M. Gabbouj, The error concealment feature in the H.26L test model, in: Proceedings IEEE International Conference on Image Processing, Rochester, NY, USA, 2002, pp. 729–732.