block loss recovery techniques for image and video communications
DESCRIPTION
Block Loss Recovery Techniques for Image and Video Communications. Jiho Park The Computational Intelligence Applications (CIA) Lab. Department of Electrical Engineering University of Washington May 21, 2002. Outline. Introduction Background for Block Recovery - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/1.jpg)
Block Loss Recovery Techniques for Image and Video Communications
Jiho Park
The Computational Intelligence Applications (CIA) Lab.
Department of Electrical Engineering
University of Washington
May 21, 2002
![Page 2: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/2.jpg)
2
Outline
Introduction
Background for Block Recovery
Spatial Block Loss Recovery Technique
Temporal Block Loss Recovery Technique
Interpolation based Coding and Data Priority
Setting
Conclusion & Future Work
![Page 3: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/3.jpg)
3
Image and Video Coding Standards
Adopt block based coding (e.g. JPEG, MPE
G, H.261/3+)
Lossy coding techniques
Images are segmented into blocks
Predictive Coding, Discrete Cosine Transform
and Quantization are adopted
![Page 4: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/4.jpg)
4
Background –Block Coding & Transmission Error
Block based coding technique
N x N block
Discrete Cosine
TransformQ RLC Packet
Encoding Packetizing Network Depacketizing DecodingImage/ Video
Source
image/video transmission system.
![Page 5: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/5.jpg)
5
Transmission & Error
During transmission, some packets are lost due to bit error, congestion of network, noise burst, or other reasons.
Packet
Packet error during transmission causes all data loss in a corresponding block.
![Page 6: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/6.jpg)
6
Background – Error Correction Automatic Retransmission Request(ARQ) – Decoder sen
ds retransmission requests to encoder when error happens. pro : error-free. cons : additional delays, unacceptable for real-time. example : TCP/IP – unreal-time internet file transfer protocol
Forward Error Correction(FEC) – Encoder inserts additional codes so that decoder can correct errors. pro : error resilient. con : additional bits
example : 18-bit FEC code in H.261(video-conferencing standard)
Error Concealment/Block Loss Recovery – Post-processing to restore damaged blocks by decoder
pro : no additional bits. con : complex decoder
![Page 7: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/7.jpg)
7
Spatial EC : Exploiting spatially neighboring image data of a missing block.
Temporal EC : Exploiting temporally adjacent frame of video.
Background –Error Concealment / Block Loss Recovery
tt-1
![Page 8: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/8.jpg)
8
Spatial Block Loss Recovery
Spatial block loss recovery is suitable for
image coding data, (e,g, JPEG, Image
based coding frame of MPEG/H.261/263+)
![Page 9: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/9.jpg)
9
Averaging the values of pixels in the same position of surrounding blocks.
Previous Research (Spatial EC) -Edge-based Average Interpolation (Ancis ’99)
1x 2x 3x
5x4x
6x 7x 8x
8
1
8
1
,, /i
k
i
jikk
jim wxwx
where wk = 0, 1, according to edges in surrounding blocks
![Page 10: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/10.jpg)
10
Previous Research (Spatial EC) - Interpolation Exploiting Interblock Correlation (Hemami ’95)
Tx
RxLx
Bx
BBRRLLTTM xwxwxwxwx
To get weight, W, the values of boundary pixels in a missing block are set to the value of adjacent known pixels.
Weight, W, are computed by solving the equations using Linear Least Squares Problem.
SXWXM
S''' XWX M
(1~64)
(1~30)
![Page 11: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/11.jpg)
11
Set Y2 to zero =>
Previous Research (Spatial EC) – DCT-based interpolation (Ziad, 2000)
,2211 XTXT
XTY
unknown.: known,: 21 XX
2
1
2221
1211
2
1
X
X
TT
TT
Y
Y
222121 XTXT0 121222
1 XTTX
Kernel DCT:T
![Page 12: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/12.jpg)
12
Previous Research – Other Algorithms
Maximally Smooth Image Recovery (Wang, ’93)
- minimizing errors between a missing block and surrounding pixels
DCT Coefficients Block Recovery (Park, ’97)
- minimizing errors between boundary pixels of a missing block and surrounding pixels
Block Recovery using POCS (Sun, ’95)
![Page 13: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/13.jpg)
13
Conventional Algorithms -
1) Not adaptive to local image structure (Ziad, Hemami,…)
2) ill-utilizing surrounding image structure (hemami,…)
3) no ability of recover certain frequency bands (Sun, Ziad, Wan
g, Park…)
Previous Research – Problem Formulation
![Page 14: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/14.jpg)
14
Block Loss Recovery – Research Goal
To design a spatial block loss recovery Technin
que More Reliable
Adaptive to Image Structure
No Retransmission
No Additional Bits for error correction
![Page 15: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/15.jpg)
15
Projections based Block Recovery – Motivation
Conventional Algorithms use information of all surrounding area. Using only highly correlated area
![Page 16: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/16.jpg)
16
Alternating Projections is projecting between two or more convex sets iteratively.
Projections onto Convex Sets –Alternating Projections
Converging to a common point
![Page 17: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/17.jpg)
17
Signal Restoration using Alternating Projections – Papoulis-Gerchberg Algorithm
Band Limiting.Convex Set, C1
IFT
Identical Middle.Convex Set, C2
FT
FT
Original Signal
Corrupted Signal
C2
![Page 18: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/18.jpg)
18
Projections based Block Recovery – Algorithm
2 Steps Pre Process : 1) Edge orientation detection
2) Surrounding vector extraction
3) Recovery vector extraction
Projections : 1) Projection operator P1
2) Projection operator P2
3) Projection operator P3
![Page 19: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/19.jpg)
19
Edge orientation in the surrounding area(S) of a missing block(M). In order to extend the geometric structure to the missing block.
Simple line masks at every i, j coordinate in surrounding area(S) of the missing block(M) for edge detection.
Pre Process 1 –Edge Orientation Detection
121
121
121
vL
111
222
111
hL
Horizontal Line Mask Vertical Line Mask
![Page 20: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/20.jpg)
20
Pre Process 1 – Edge Orientation Detection
Responses of the line masks at window W :
Total magnitude of responses :
Th > Tv ; Horizontal line dominating area
Th < Tv ; Vertical line dominating area
987
654
321
www
www
www
W987654321 w-w-w-w2w2w2w-w--w hR
987654321 w-w2w-w-w2w-w-w2-w vR
,||T S
hh R S
vv R ||T
![Page 21: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/21.jpg)
21
Pre Process 2 – Surrounding Vectors
Surrounding Vectors, sk, are extracted from surrounding area of a missing block by N x N window.
Each vector has its own spatial and spectral characteristic. The number of surrounding vectors, sk, is 8N.
}W),(),,(:{ jijixxks
![Page 22: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/22.jpg)
22
Pre Process 3 – Recovery Vector Recovery vectors are extracted to restore missing pixels. Two positions of recovery vectors are possible according to the
edge orientation.
Recovery vectors consist of known pixels(white color) and missing pixels(gray color).
The number of recovery vectors, rk, is 2.
}W),(),,(:{ jijixxkr
Vertical line dominating area Horizontal line dominating area
![Page 23: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/23.jpg)
23
Projections based Block Recovery –Projection operator P1
Recovery vectors, ri, for i = 1, 2
Surrounding vectors, sj , for j = 1 ~ 8N
Surrounding vectors, S, form a convex hull in N2-dimensional space
Recovery vectors, R, are orthogonally projected onto the line defined by the closest surrounding vector, si, j : Projection Operator P1.
![Page 24: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/24.jpg)
24
Projections based Block Recovery –Projection operator P1
Projection operator P1 :
Convex hull (formed by surrounding vectors, containing information of local image structure)
![Page 25: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/25.jpg)
25
Projections based Block Recovery –Projection operator P1
Surrounding vectors, sj , for j = 1 ~ 8N Recovery vectors, ri, for i = 1, 2
The closest vertex, sdi , from a recovery vector, ri.
or equivalently in DCT domain,
P1 :
Njiford jij
i 81,21||||minarg sr
Njiford jij
i 81,21||||minarg SR
21,||||
,)(
2
ii
i
d di
idiiP S
R
RSRS
![Page 26: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/26.jpg)
26
Convex set C2 acts as an “identical middle”.
Projection operator P2 :
Projections based Block Recovery –Projection operator P2
otherwise
nforFFC
o
n
ff
ff
:
L: maxmin2
otherwise
nFforF
nFforF
P
n
n
n
n
f
f
f
f L
L
max,max
min,min
2
![Page 27: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/27.jpg)
27
Convex set C3 acts as a convex constraint between missing pixels and adjacent known pixels, (fN-1 fN).
where, and
is a N x N recovery vector in column vector form.
Projections based Block Recovery – Projection operator P3
fN-1 fN
}||:{3 EC n gg
)}(....,),{( ,,10,0,1 NNNNNN ffffg }....,,,{ 21 Nffff
![Page 28: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/28.jpg)
28
Projections based Block Recovery – Projection operator P3
Projection operator P3 :
otherwise
nEforE
nEforE
P
mn
nmn
nmn
mn
,
,1
,1
,3 L,
L,
f
gf
gf
f
![Page 29: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/29.jpg)
29
Projections based Block Recovery –Iterative Algorithm
Missing pixels in recovery vectors are restored by iterative algorithm of alternating projections :
N x N windows moving :
ii fPPPf 3211
Vertical line dominating area Horizontal line dominating area
![Page 30: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/30.jpg)
30
Projections based Block Recovery - Summary
Edge Orientation Detection
Surrounding Vector Extraction
Recovery Vector Extraction
Projection Operator P1
Projection Operator P2
Projection Operator P3
Iteration=I?
All pixels?
![Page 31: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/31.jpg)
31
Simulation Results – Test Data and Error
512 x 512 “Lena”, “Masquerade”, “Peppers”, “Boat”, “Elaine”, “Couple”
176 x 144 “Foreman” 352 x 240 “Flower Garden”
8 x 8 pixel block loss 16 x 16 pixel block loss 8 x 8 consecutive block losses
Peak Signal – Noise – Ratio
)|),(ˆ),(|
255log(10
1 1
2
2
N
i
M
j
jixjix
MNPSNR
![Page 32: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/32.jpg)
32
Simulation Results –Lena, 8 x 8 block loss
Original Image Test Image
![Page 33: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/33.jpg)
33
Simulation Results –Lena, 8 x 8 block loss
Ancis, PSNR = 28.68 dB Hemami, PSNR = 31.86 dB
![Page 34: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/34.jpg)
34
Simulation Results –Lena, 8 x 8 block loss
Ziad, PSNR = 31.57 dB Proposed, PSNR = 34.65 dB
![Page 35: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/35.jpg)
35
Simulation Results –Lena, 8 x 8 block loss
Ancis
PSNR = 28.68 dB
Hemami
PSNR = 31.86 dB
Ziad
PSNR = 31.57 dB
Proposed
PSNR = 34.65 dB
![Page 36: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/36.jpg)
36
Simulation Results – Each StepLena 8 x 8 block loss
(a)
(b)
(c)
![Page 37: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/37.jpg)
37
Simulation Results –Peppers, 8 x 8 block loss
Original Image Test Image
![Page 38: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/38.jpg)
38
Simulation Results – Peppers, 8 x 8 block loss
Ancis, PSNR = 27.92 dB Hemami, PSNR = 31.83 dB
![Page 39: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/39.jpg)
39
Simulation Results – Peppers, 8 x 8 block loss
Ziad, PSNR = 32.76 dB Proposed, PSNR = 34.20 dB
![Page 40: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/40.jpg)
40
Simulation Results – PSNR (8 x 8)
Lena Masqrd Peppers Boat Elaine Couple
Ancis 28.68 25.47 27.92 26.33 29.84 28.24
Sun 29.99 27.25 29.97 27.36 30.95 28.45
Park 31.26 27.91 31.71 28.77 32.96 30.04
Hemami 31.86 27.65 31.83 29.36 32.07 30.31
Ziad 31.57 27.94 32.76 30.11 31.92 30.99
Proposed 34.65 29.87 34.20 30.78 34.63 31.49
![Page 41: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/41.jpg)
41
Simulation Results –Lena, 8 x one row block loss
Original Image Test Image
![Page 42: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/42.jpg)
42
Simulation Results –Lena, 8 x one row block loss
Hemami, PSNR = 26.86 dB Proposed, PSNR = 30.18 dB
![Page 43: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/43.jpg)
43
Simulation Results –Masquerade, 8 x one row block loss
Original Image Test Image
![Page 44: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/44.jpg)
44
Simulation Results –Masquerade, 8 x one row block loss
Hemami, PSNR = 23.10 dB Proposed, PSNR = 25.09 dB
![Page 45: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/45.jpg)
45
Simulation Results –Lena, 16 x 16 block loss
Original Image Test Image
![Page 46: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/46.jpg)
46
Simulation Results –Lena, 16 x 16 block loss
Ziad, PSNR = 28.75 dB Proposed, PSNR = 32.70 dB
![Page 47: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/47.jpg)
47
Simulation Results –Foreman, 16 x 16 block loss
Original Image Test Image
Ziad, PSNR = 25.65 dB Proposed, PSNR = 30.34 dB
![Page 48: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/48.jpg)
48
Simulation Results –Flower Garden, 16 x 16 block loss
Original Image Test Image
Ziad, PSNR = 20.40 dB Proposed, PSNR = 22.62 dB
![Page 49: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/49.jpg)
49
Simulation Results – PSNR (Row, 16 x 16)
(16 x 16) Lena Foreman Garden
Ziad 28.75 25.65 20.40
Proposed 32.70 30.34 22.62
(8 x Row) Lena Maskrd Peppers Boat Elaine Couple
Hemami 26.86 23.10 25.41 24.54 26.87 24.30
Proposed 30.18 25.09 28.31 26.06 30.11 26.12
![Page 50: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/50.jpg)
50
Simulation Results –Mean & Variance of Error (8 x 8)
,|ˆ|mean Br
xx 22mean|ˆ|variance
Br
xx
Lena Maskrd Peppers Boat Elaine Couple
m v m v m v M v m v m V
Ancis 11.5 242.2 18.6 437.9 12.4 290.9 15.9 392.2 11.5 154.2 12.2 264.2
Sun 12.6 117.2 15.6 276.2 12.3 126.6 16.3 241.1 11.8 81.6 14.4 186.7
Park 8.2 138.7 13.4 267.5 7.9 123.1 11.4 236.4 7.8 76.9 10.0 172.4
Hemami 7.2 127.4 12.7 313.3 7.7 121.2 10.6 208.9 8.7 94.8 8.4 185.7
Ziad 7.3 137.8 12.3 290.7 7.4 91.3 10.0 170.4 9.4 88.5 7.3 165.1
Proposed 5.4 64.8 10.0 184.3 5.9 69.3 9.1 147.4 6.8 47.1 7.2 143.6
![Page 51: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/51.jpg)
51
Projections based Spatial Block Recovery – Discussion
New projection based block recovery algorithm was
presented.
8 images and 3 error cases were tested.
5 existing block recovery algorithms were used for
comparison.
The proposed algorithm outperforms other techniques
in all error cases of all images.
![Page 52: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/52.jpg)
52
Temporal Block Loss Recovery
In video coding (e, g, MPEG, H.261/3+),
temporal recovery is more effective.
tt-1
![Page 53: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/53.jpg)
53
Background – MPEG, H.261/3+
Standards for video coding Block and predictive coding techniques are
adopted.
I B B BP B P B B
Group of pictures
P B B
![Page 54: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/54.jpg)
54
Predictive Coding : Eliminating the inter-pixel redundancies
Background – Predictive Coding
- =Previous FrameCurrent Frame
MV(x, y) = (x0 – x1, y0 – y1)
(x0 , y0) (x1 , y1)
MV(x, y) Redundancy
MPEG/H26x bitstream
![Page 55: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/55.jpg)
55
Errors in video streams during transmissions
Background – Error in MPEG
MV(x, y) Redundancy
MV(x, y) Redundancy
MV(x, y) Redundancy
![Page 56: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/56.jpg)
56
Motion Vector Recovery Techniques
Motion Flow Estimation Method
Spatial Correlation Exploit Method
Temporal Correlation Exploit Method
Previous Research – Motion Vector Recovery
![Page 57: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/57.jpg)
57
Zero Motion Vector (ZM)
Average of Surrounding Motion Vectors (AV)
Previous Research - Motion Flow Estimation
MV(x, y) = (0, 0)
Simplest but poor performance
Simple but poor performance
Smn
mn yxMVyxMV,
6/),(),(
![Page 58: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/58.jpg)
58
Motion Flow Interpolation (MFI) (IEEE 1999)
Previous Research - Motion Flow Estimation
Pro : Simple
Con : different Motion Vectors in the same block
N
vyvyNvxvxNyxd
BnTnRnLn
nn
2
)()(),(
![Page 59: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/59.jpg)
59
Boundary Matching Algorithm (BMA) (IEEE 1993)
Previous Research –Spatial Correlation Exploit Method
),1,(),,( 00
10
0
nyxfnyxfC RC
x
xx
A
N
),,1(),,( 000
10
0
nyxfnyxfC RC
y
yy
L
N
),,(),1,( 000
10
0
nNyxfnNyxfC RC
x
xx
B
N
)(minarg,,
LBAxx DDDmvmvyx
A
L
B
![Page 60: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/60.jpg)
60
Decoder Motion Vector Estimation (DMVE) (IEEE 2000)
Previous Research –Temporal Correlation Exploit Method
),,(),,(11 0
0
0
0
nyxfnyxfC RC
y
wyy
x
xx
A
N
)(minarg,,
LBAxx DDDmvmvyx
),,(),,(11 0
0
0
0
nyxfnyxfC RC
wy
yy
x
xx
B
N
N
N
),,(),,(11 0
0
0
0
nyxfnyxfC RC
wy
wyy
x
wxx
L
N
A
L
B
![Page 61: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/61.jpg)
61
Forward-Backward Block Matching (FB BM) (IEEE 2000)
Previous Research –Temporal Correlation Exploit Method
)),,()1,,((minarg,11 0
0
0
0
,njifnjyixfmvmv
y
yj
x
xiyx
AyAx
N
N
)),,()1,,((minarg,121 0
0
0
0
,njifnjyixfmvmv
N
N
N y
yj
x
xiyx
ByBx
),,()1,,((minarg,11 0
0
0
0
,njifnjyixfmvmv
y
yj
x
xiyx
CyCx
N
N
)),,()1,,(121 0
0
0
0
njifnjyixfN
N
N y
yj
x
xi
}}{,,,,,arg{min, , yxCyCxByBxAyAxyx Dmvmvmvmvmvmvmvmv
![Page 62: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/62.jpg)
62
Advantage - Good Performance
Disadvantage -
1) Not Adaptive to Local Image Structure
2) Not Resilient to Propagation Error
Previous Research – Problem Formulation
Temporal/Spatial Correlation Exploit Methods
![Page 63: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/63.jpg)
63
Temporal Block Loss Recovery – Research Goal
To design a Temporal Restoration Technique More reliable
Adaptive to local image structure
Adaptive/Resilient to propagation error
Utilizing Projections
No retransmission
No additional bits for error correction
![Page 64: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/64.jpg)
64
3 Steps
Pre Process – Edge extension using Projections P1, P2
Motion vector recovery - 1) Local image structure estimation
2) Propagation error
estimation
Post process - Spatial Compensation using Projections P3
Adaptive Temporal Block Recovery – Algorithm
![Page 65: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/65.jpg)
65
Vertical edge detection using vertical line mask
on SN and SS
Temporal Block Loss recovery –Edge extension with projections P1, P2
,||T NS
vnv R ,||T sS
vsv R Total Magnitude :
121
121
121
vL
Vertical Line Mask
54321 w2w-w-w2-w vR9876 w-w2w-w-
![Page 66: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/66.jpg)
66
Adaptive Motion Vector Recovery 1 –Edge Extension with projections P1, P2
If , extend edge by projecting recovery pixels to above
and below reference blocks (we assume there is strong edge in thr
ee blocks.)
csvnv TT ,
otherwise
nforFFC
o
n
ff
ff
:
L: maxmin2
otherwise
nFforF
nFforF
P
n
n
n
n
f
f
f
f L
L
max,max
min,min
2
21,||||
,)(
2
ii
i
d di
iiiP S
SS R
R
RRRP1:
![Page 67: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/67.jpg)
67
Adaptive Motion Vector Recovery –Distortion Metric
},),,(:{ 0000 NyjyxiKxjixxfn
},),,(:{ 0000 NyjyVNxiNxjixxfs
},),,(:{ 0000 yjLyNxixjixxfw
},),,(:{ 0000 NyjyVxixjixxfmt
},),,(:{ 0000 NyjyNxiVNxjixxfmb
![Page 68: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/68.jpg)
68
),,(),,( 0000,,,
1 1
Rjijiji NjyyixxfNjyixfD n
K
i
N
j
nn
),,(),,( 0000,,,
1 1
Rjijiji NjyyiNxxfNjyiNxfD s
K
i
N
j
ss
),,(),,( 0000,,,
1 1
Rjijiji NjyyixxfNjyixfD n
V
i
N
j
nmt
),,(),,( 0000,,,
1 1
Rjijiji NjyyiNxxfNjyiNxfD n
V
i
N
j
nmb
),,(),,( 0000,,,
1 1
Rjijiji NjyyixxfNjyixfD n
N
i
L
j
nw
)(minarg,,
wmbmtsnxx DDDDDmvmvyx
Adaptive Motion Vector Recovery –Distortion Metric (Temporal Correlation)
![Page 69: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/69.jpg)
69
: Weight for Local Image Structure
: Weight for Local Image Correlation
Adaptive Motion Vector Recovery –Weights in the Distortion Metric
ji ,ji , : Weight for Propagation Error
Three weights enable the proposed algorithm to be adaptive to local image structure and propagation error
ji ,
![Page 70: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/70.jpg)
70
Adaptive Motion Vector Recovery 1 –Weight i,j for Local Image Structure
v
vjisjin
Tround ,,,, ,
2
)(1
,,,,,,
jisjinjiw
1,0 ,,,, jisjin
2
)( ,,,, jisjin
jiw ,,1
1
,
,||T NS
vnv R ,||T sS
vsv R
![Page 71: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/71.jpg)
71
Adaptive Motion Vector Recovery 1 –Weights i,j for Local Image Correlation
/)1(,
dji ec
ji,
d
![Page 72: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/72.jpg)
72
Adaptive Motion Vector Recovery 2 – Propagation Error Estimation for Weights i,j
Error Propagation and Error States
![Page 73: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/73.jpg)
73
Adaptive Motion Vector Recovery 2 –Weights i,j for Propagation Error
Error States of Pixels
Sc(i,j) = 0, when correctly received
Sres(i,j) = Sref(i,j)+1, when restored
Weight, ji ,
10,1 ,,, jijiji S
1
ji ,
1
S (i,j)
![Page 74: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/74.jpg)
74
Adaptive Motion Vector Recovery – Weights and Motion Vector
)(minarg,,
wmbmtsnxx DDDDDmvmvyx
)( ),(),(),(),(),( mvxmvxwmvxmvxmbmvxmvxmtmvxmvxsmvxmvxn DDDDDe
/)1(,
dji ec
v
vjisjin
Tround ,,,, ,
2
)(1
,,,,,,
jisjinjiw
10,1 ,,, jijiji S
![Page 75: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/75.jpg)
75
if , apply spatial constraint
Temporal Block Loss Recovery –Spatial Compensation using Projection P3
eTe
otherwise
nEforE
nEforE
P
mn
nmn
nmn
mn
,
,1
,1
,3 L,
L,
f
gf
gf
f
where, andis a N x N recovery vector in column vector form.
)}(....,),{( ,,10,0,1 NNNNNN ffffg }....,,,{ 21 Nffff
![Page 76: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/76.jpg)
76
Adaptive Temporal Block Loss Recovery – Summary
Local Image Structure Estimation
Propagation Error Estimation
Motion Vector Recovery
Spatial Constraints, P3
e > Te
End
Edge Extension, P1, P2
![Page 77: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/77.jpg)
77
352 x 240 “Flower Garden”, “Table Tennis”, “Mobile”, “Football”
176 x 144 “Foreman”
Simulation Results – Test Data and Parameters
K = 3, L = 1, V = 1. (size of surrounding area)
=v= 3500 ()
= 1.1 if d=1, = 1.0 else = 0.1 ()
Tc=7000 (edge extension)
Te=1800 (spatial compensation)
![Page 78: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/78.jpg)
78
Simulation Results –Errors
I B P P P
- Error in every B & P frame
- 11 / 12 frame error (91.7 % frame error)
. . .
. .
.
B B BB B B B
I B P P PB B BB B B B
![Page 79: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/79.jpg)
79
Simulation Results – Flower Garden
Original Sequence Test Sequence
![Page 80: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/80.jpg)
80
Simulation Results – Flower Garden
ZM, PSNR = 16.15 dB AV, PSNR = 18.64 dB
![Page 81: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/81.jpg)
81
Simulation Results – Flower Garden
MFI, PSNR = 19.29 dB BMA, PSNR = 19.83 dB
![Page 82: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/82.jpg)
82
Simulation Results – Flower Garden
FB BM, PSNR = 19.21 dB Proposed, PSNR = 20.71 dB
![Page 83: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/83.jpg)
83
Simulation Results – Flower Garden
![Page 84: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/84.jpg)
84
Simulation Results – Foreman
Original Sequence Test Sequence
ZM PSNR = 24.71 dB AV PSNR = 26.22 dB
![Page 85: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/85.jpg)
85
Simulation Results – Foreman
MFI PSNR = 27.09 dB BMA PSNR = 28.76 dB
FB BM PSNR = 27.46 dB Proposed PSNR = 29.82 dB
![Page 86: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/86.jpg)
86
Simulation Results – Foreman
![Page 87: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/87.jpg)
87
Simulation Results – Average PSNR
Garden Tennis Football Mobile Foreman
MV 16.15 22.40 18.06 17.49 24.71
AV 18.64 21.98 18.72 19.03 26.22
BMA 19.83 23.55 19.41 19.75 28.76
DMVE 19.88 24.04 19.64 20.02 28.77
MFI 19.29 22.77 19.29 19.60 27.09
F-B BM 19.21 22.49 19.05 19.59 27.46
Proposed 20.71 24.52 20.32 20.66 29.82
![Page 88: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/88.jpg)
88
Adaptive Temporal Block Loss Recovery – Discussion
New adaptive temporal block loss recovery algorithm
was presented.
5 sequences were tested.
6 existing block recovery algorithms were used for
comparison.
The proposed algorithm outperforms other techniques
in all sequences.
![Page 89: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/89.jpg)
89
Interpolation based Coding and Data Priority Setting Technique
Block loss recovery technique only by decode part
does not provide a good error resilient performance
Inter-active method between encoder and decoder is
more useful for image/video communication.
Data priority setting problem using the proposed
spatial block loss recovery.
![Page 90: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/90.jpg)
90
Interpolation based Coding and Data Priority Setting – Problem Formulation
Many encoders have a function that sets priorities to image/video data for transmission.
When network congestions happen, the network discards low priority data and preserves high priority data.
Coding standards set high priority to low frequency data and low priority to high frequency data.
It does not consider block loss recovery ability of decoders during priority setting.
Packet Packet. . . . . .PacketL H H
![Page 91: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/91.jpg)
91
Interpolation based Coding and Priority Setting Method – Research Goal
To make coding system be able to give priorities to
image data by considering decoder’s block loss
recovery ability
To outperform compression rates of JPEG
JPEG compatible
Compatible to the proposed spatial block recovery
algorithm
![Page 92: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/92.jpg)
92
Interpolation based Coding - Motivation
If the block loss recovery algorithm in encoder restores missing blocks faithfully, then those blocks can be set as low priority blocks and discarded during network congestions, or not need be transmitted by sender.
Original Point
Compressed Point by JPEG
Interpolated Point by projections
distance(A-B) < distance(A-C) Low priority distance(A-B) > distance(A-C) High priority
![Page 93: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/93.jpg)
93
Interpolation based Coding – Algorithm 1
Compute PSNRj & Bit Rate(R
b) of every block
compressed image/Video
Restore every block by the proposed algorithm. and Compute PSNRr of every restored b
lock
Sort blocks by distortion & bit rate.
((PSNRr - PSNRj) * Rb)
Set high priority to or Store/Transmit only the block of low distortion & low bit rate
![Page 94: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/94.jpg)
94
Interpolation based Coding – Result 1
JPEG Coding
PSNR = 32.27 dB
Size = 0.30 BPP = 9,902 Byte
w/ Removed Blocks
Blocks : 447 / 4096 = 11%
Size = 0.29 BPP
I-based Coding
PSNR = 32.35 dB
Size = 0.29 BPP = 9,634 Byte
![Page 95: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/95.jpg)
95
Interpolation based Coding – Algorithm 2
We applied algorithm1 and setting priorities to blocks
compressed image/Video
Investigate which surrounding blocks are used for the
recovery of low priority blocks
Set highest priority to the used surrounding blocks
Apply algorithm 1 to unused surrounding blocks
Set second lowest priority to some blocks with the results
![Page 96: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/96.jpg)
96
Interpolation based Coding – Result 2
JPEG Coding
PSNR = 32.27 dB
Size = 0.30 BPP = 9,902 Byte
w/ Removed Blocks
Blocks : 557 / 4096 = 14%
Size = 0.27 BPP
I-based Coding
PSNR = 32.37 dB
Size = 0.27 BPP = 9,570 Byte
![Page 97: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/97.jpg)
97
Interpolation based Coding – Result 2
JPEG Coding
PSNR = 32.27 dB
Size = 0.30 BPP = 9,902 Byte
I-based Coding
PSNR = 32.37 dB
Size = 0.27 BPP = 9,570 Byte
20% Packet Drop
PSNR = 29.70 dB
Size = 0.30 BPP = 9,902 Byte
![Page 98: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/98.jpg)
98
Interpolation based Coding - Result
PSNR vs. BPP in Lena Image
![Page 99: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/99.jpg)
99
Interpolation based Coding - Result
PSNR vs. BPP in Peppers Image
![Page 100: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/100.jpg)
100
Interpolation based Coding – Discussion
New interpolation based image coding algorithm was presented.
Priority data setting by encoder using decoder’s block loss recovery ability
JPEG baseline coding technique was used for comparison
The proposed coding algorithm outperforms JPEG coding on different bit rates.
New priority setting using proposed spatial block loss recovery algorithm was presented
![Page 101: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/101.jpg)
101
Conclusion Spatial block recovery
New method of exploiting and utilizing highly correlated surrounding area to a missing block is developed.
New alternating projections based algorithm is proposed. Proposed algorithm outperforms existing image block
recovery techniques.
Temporal block recovery Adaptive motion vector recovery is proposed. Edge extension for MV recovery is introduced. Adaptive spatial compensation is developed. Proposed algorithm outperforms existing video recovery
techniques.
![Page 102: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/102.jpg)
102
Conclusion Interpolation-based low-bit-rate coding
Decoder based data priority setting is presented.
Active error resilient method is presented.
Presented coding technique outperforms JPEG standard.
![Page 103: Block Loss Recovery Techniques for Image and Video Communications](https://reader035.vdocument.in/reader035/viewer/2022070406/56814056550346895dabc9a1/html5/thumbnails/103.jpg)
103
Future Work
Fast search method for the proposed algorithms will
reduce search time and can be implemented in the
system requiring fast computation.
FEC technique for the proposed algorithms will
improve image/video communication reliability.