improving frame interpolation with spatial motion...
TRANSCRIPT
Improving Frame Interpolation With Spatial Motion Smoothing For Pixel Domain
Distributed Video Coding
João [email protected]
Instituto Superior de Engenharia de Lisboa Instituto de Telecomunicações
Portugal
EC–SIP–M 200501/07/05
Summary
1. Context and Applications
2. Objectives
3. IST-PDWZ Video Codec
4. Frame Interpolation
5. Experimental Results
6. Conclusions and Future Work
EC–SIP–M 200501/07/05
Traditional Video Coding
– Predictive coding framework:– Encoder is 5 to 10 times more complex than decoder mainly due to
the motion estimation/compensation tools.– Well-suited for “one-to-many” topologies:
– Broadcasting or video-on-demand applications.
EC–SIP–M 200501/07/05
Emerging Applications, New Requirements
– Applications:– Mobile video.– Multimedia sensor networks.– Wireless video surveillance.– Multi-view acquisition.– …
– Encoding requirements:– Low complexity.– Low power-consumption.– Low cost.– ...
EC–SIP–M 200501/07/05
Need for a New Coding Paradigm
– Coding configuration with low-complexity and low-power encoder at the expense of a high-complexity decoder– Without compromising coding efficiency.
New video coding paradigm:
Distributed Video Coding (DVC) Distributed Video Coding (DVC) Exploits video statistics, partially or totally, at the decoder.
EC–SIP–M 200501/07/05
Summary
1. Context and Applications
2. Objectives
3. IST-PDWZ Video Codec
4. Frame Interpolation
5. Experimental Results
6. Conclusions and Future Work
EC–SIP–M 200501/07/05
Main Goal
The challenge is:
How to generate the best side information (a frame) as close as How to generate the best side information (a frame) as close as
possible to the current frame to be decoded ? possible to the current frame to be decoded ?
Distributed videodecoder
Lightweight encoder
Side Information
The rate-distortion (RD) performance of a distributed video coding scheme is highly dependent on the quality of the side information.
EC–SIP–M 200501/07/05
Summary
1. Context and Applications
2. Objectives
3. IST-PDWZ Video Codec
4. Frame Interpolation
5. Experimental Results
6. Conclusions and Future Work
EC–SIP–M 200501/07/05
IST Pixel-Domain Wyner-Ziv Video Codec
– Based on the Wyner-Ziv coding scenario.– Intra-frame encoder and inter-frame decoder.– X source: even frames of the video sequence.– Y source corresponds to the Side Information:
– Generated using the odd frames at the decoder – Y is an estimate of the current frame X and is available at the decoder.
– Channel coding of X allows to improve the quality of Y.
Inter-framedecoder
Intra-frame encoder
Virtual channeldependence
Side Information
EC–SIP–M 200501/07/05
IST Pixel-Domain Wyner-Ziv Video Codec
– Turbo code based Slepian-Wolf codec.– Each bitplane is independently turbo coded.– Two frame types:
– Key-frames (odd frames) and Wyner-Ziv frames (even frames).– Side information (Y2i) is obtained by frame interpolation.
Wyner-Ziv Encoder Wyner-Ziv Decoder
Slepian-Wolf Encoder Slepian-Wolf Decoder
2N- Level Uniform
Quantizer
X2i ..
..
Bitplane 1X'2i
Y2i
X2i - 1
X2i + 1
Wyner-Ziv Bitstream
Turbo Encoder
Turbo Decoder Reconstruction
FrameInterpolation
Buffer
Feedback Channel
Bitplane N
EC–SIP–M 200501/07/05
Slepian-Wolf Codec
– Turbo encoder:– Two identical recursive systematic
convolutional (RSC) encoders.– Pseudo-random interleaver.
– Each RSC encoder outputs the systematic and the parity streams:– Systematic stream is discarded.– Parity stream is stored in the buffer
and punctured.– Decoder request parity bits until
successful decoding.
– Iterative turbo decoder:– Two SISO decoders. – Maximum A Posteriori (MAP) algorithm.– Laplacian distribution to model the correlation between X and Y.
EC–SIP–M 200501/07/05
Summary
1. Context and Applications
2. Objectives
3. IST-PDWZ Video Codec
4. Frame Interpolation
5. Experimental Results
6. Conclusions and Future Work
EC–SIP–M 200501/07/05
Frame Interpolation: Architecture
– Block-based motion compensated interpolation (5 steps):– 1) Low pass filter to improve the motion vectors reliability.– 2) Forward motion estimation.– 3) Bi-directional motion refines initial motion vector estimate.– 4) Spatial smoothing of motion vectors.– 5) Bi-directional MC to fill the interpolated frame.
EC–SIP–M 200501/07/05
Initial MV & Bi-directional ME
– Motion vectors obtained by FME are candidates for each non-overlapped block in the interpolation frame:– Selected the MV that intercepts the interpolated frame closest to the
center of block.– Find linear trajectory (symmetric MVs) between key frames passing at the
center of the block in the interpolated frame:– Small displacement around the initial block position.
EC–SIP–M 200501/07/05
Spatial Motion Smoothing
– Spatial smoothing algorithms target: – Reduction on the number of false motion vectors when compared to
the true motion field.– Better spatial homogeneity of the resulting motion field.
– Weighted vector median filters are proposed:– Filter smoothing strength depends on the prediction MSE.– Low value weights when MSE for the candidate vector is high.– High value weights when MSE for the candidate vector is low.
1 1
N N
j wvmf j j i jL Lj j
w x x w x x= =
− ≤ −∑ ∑ ( )( )
,,
cj
j
MSE x Bw
MSE x B=
EC–SIP–M 200501/07/05
Spatial Motion Smoothing
– Without motion vector smoothing filter
EC–SIP–M 200501/07/05
Spatial Motion Smoothing
– With motion vector smoothing filter
EC–SIP–M 200501/07/05
Summary
1. Context and Applications
2. Objectives
3. IST-PDWZ Video Codec
4. Frame Interpolation
5. Experimental Results
6. Conclusions and Future Work
EC–SIP–M 200501/07/05
Coastguard Sequence
Coastguard sequence
26.5
28.5
30.5
32.5
34.5
36.5
38.5
40.5
42.5
0 50 100 150 200 250 300 350 400 450 500 550 600Rate of even frames (kbps)
PSN
R o
f eve
n fr
ames
(dB
)
H.263+ Intra
H.263+ I-B-I-B
PDWZ Average
PDWZ FME
PDWZ BiME
PDWZ SS
11.5 dB
EC–SIP–M 200501/07/05
Foreman Sequence
Foreman sequence
27.0
29.0
31.0
33.0
35.0
37.0
39.0
41.0
43.0
0 50 100 150 200 250 300 350 400 450 500 550 600 650Rate of even frames (kbps)
PSN
R o
f eve
n fr
ames
(dB
)
H.263+ IntraH.263+ I-B-I-BPDWZ AveragePDWZ FMEPDWZ BiMEPDWZ SS
10 dB
EC–SIP–M 200501/07/05
Rate-Distortion Comparison
Foreman QCIF Sequence @ 30fps
35.0
36.0
37.0
38.0
39.0
40.0
41.0
42.0
43.0
0 50 100 150 200 250 300 350 400
Rate of even frames [kbps]
PS
NR
of e
ven
fram
es [d
B]
H.263+ I-B-I-BSpatial Motion SmoothingStanford
2.6 dB
2 dB
EC–SIP–M 200501/07/05
Summary
1. Context and Applications
2. Objectives
3. IST-PDWZ Video Codec
4. Frame Interpolation
5. Experimental Results
6. Conclusions and Future Work
EC–SIP–M 200501/07/05
Conclusions & Future Work
– New motion compensated frame interpolation tools are proposed and compared in a DVC framework.
– Major gains in RD performance:– Essentially due to spatial motion smoothing.
– Future Work:– Adaptation to longer GOPs.– Improve performance when large camera movements occur.– Iterative motion refinement approach using decoded image and
keyframes.
Thanks for your attention !