overlapping in video signal processingkatto/conferences/other/icccs... · (ieee trans csvt-94 &...
TRANSCRIPT
Katto lab.1
Overlapping in Video Signal Overlapping in Video Signal ProcessingProcessing
Jiro Katto
Dept. of Computer Science, Waseda University
E-Mail: [email protected]
2
Katto lab.
Outline
Introduction“Overlapped” Transform“Overlapped” Motion Compensation“Overlapped” PredictionDenoising by “Overlapping”Conclusions
3
Katto lab.
What I am ...
90 95 00 05
Ph.D Student Company University
Image/Video Compression
Multimedia Communication Systems
Multimedia Signal Processing
MPEG, ITU-T
4
Katto lab.
What I did ...
90 95 00 05
Student Company University
Image/Video Compression
Multimedia Communication Systems
Multimedia Signal Processing
MPEG, ITU-T
Overlapped MC(IEEE Trans CSVT-94 & IEEE ICASSP-95)
IPB Prediction & Rate Control
(IEEE ICIP-95)
Wavelet Image Coding (SPIE VCIP-91)
Denoising using Motion Estimation & Pixel Shift(IEEE ICASSP-2008)
5
Katto lab.
Transform Quantization
InverseQuantization
InverseTransform
+
−
FrameMemory
MotionCompensation
MotionEstimation
EntropyCoding
Rate Control
input
local decoder
+
MemoryMotionCompensation
EntropyDecoding
InverseQuantization
InverseTransform
output
Encoder Decoder
Video Coder
OverlappedTransform
OverlappedTransform
OverlappedMotion
Compensation
OverlappedMotion
CompensationBi-directional(Overlapped)Prediction
Bi-directional(Overlapped)Prediction Denoising by
OverlappingDenoising byOverlapping
6
Katto lab.
1. “Overlapped” Transform
J.Katto and Y.Yasuda: "Performance Evaluation of Subband Coding and Optimization of Its Filter Coefficients", SPIE VCIP'91, Nov.1991.
Block TransformDCT (Discrete Cosine Transform)DFT (Discrete Fourier Transform)KLT (Karhunen-Loeve Transform)
Overlapped TransformWavelet transform / Subband codingLOT (Lapped Orthogonal Transform)MDCT (Modified DCT)
7
Katto lab.
Wavelet Transform
2h0(n)x(n)
2h1(n)
2h0(n)
2h1(n)
2h0(n)
2h1(n)
LLL
LLH
LH
H2分割フィルタバンクのツリー接続
π
角周波数
0
LLL LLH LH HLL
H
LH
LPF
HPF tree structured filter bank
frequency
8
Katto lab.
Problems (at that time)
DCT was widely used in image/video compression (JPEG/MPEG) due to its suboptimal compression capability, but suffers from blocking artifacts.Wavelet transform is a promising alternative because it can alleviate the blocking artifacts due to its overlapping properties. However, there was no theoretical compression performance measure of Wavelet transform.
original JPEG
9
Katto lab.
Optimum Bit Allocation Problem
N.S.Jayant and P.Noll: “Digital Coding of Waveforms”, Prentice Hall, 1984.
minimize reconstruction error variance,
under constant bit rate constraint
Methods: e.g. Lagrange multiplier method
( ))( 22qr f σσ =
constR ≤
quantization error variance
10
Katto lab.
Coding Gain
N.S.Jayant and P.Noll: “Digital Coding of Waveforms”, Prentice Hall, 1984.
NN
kk
N
kk
TCNG 1
1
0
2
1
0
21
⎟⎟⎠
⎞⎜⎜⎝
⎛=
∏
∑−
=
−
=
σ
σN ×N matrix(orthogonal)
N ×N matrix(orthogonal)
optimum bit allocationproblem
“Coding Gain”σk
2: coefficient varianceTC : transform coding
SIZE (N)
3
4
5
6
7
8
9
10
11
0 2 4 6 8 10 12 14 16
GA
IN (
dB)
OPTIMUM (ρ=0.95)
KLT, DCT
DFT
ρ : correlation factor
[Theory] DCT can approximate KLT when ρ → 1
11
Katto lab.
Coding Gain of Wavelet Transform ?
multirate filter bank
optimum bit allocationproblem
∏−
=⎟⎟⎠
⎞⎜⎜⎝
⎛=
1
0
1K
k k
kk
wavelet kBAG α
α
2h0(n)x(n)
2h1(n)
2h0(n)
2h1(n)
2h0(n)
2h1(n)
LLL
LLH
LH
H2分割フィルタバンクのツリー接続
α0
α1
αK-1
tree-structured filter bank
22xky A
kσσ ⋅=
∑−
=
⋅=1
0
22K
kqkr k
B σσ
{ })(nhk←
{ })(ngk←
sampling ratio
12
Katto lab.
Advantages & Evaluation Example
Proposed coding gain includes classical coding gain as its special case, and enables integrated performance comparison with block transform.Proposed coding gain can be applied to non-orthogonal filter bank (e.g. bi-orthogonal case).
provides theoretical validity on JPEG-2000’s 9x7 filter (bi-orthogonal filter)
13
Katto lab.
2. “Overlapped” Motion Compensation
J.Katto, J.Ohki, S.Nogaki and M.Ohta: "Wavelet Codec with Overlapped Motion Compensation for Very Low Bit Rate Environment", IEEE Trans. on Circuit & Systems for Video Technology, June.1994.J.Katto and M.Ohta: "An Analytical Framework for Overlapped Motion Compensation", IEEE ICASSP'95, May.1995.
(Gradient Method: Optical Flow)Block Motion Compensation
16x16 macroblock (until MPEG-4)4x4 ~ 16x16 macroblocks (H.264/AVC)
Overlapped Motion Compensation32x32 macroblock(Motion estimation may use 16x16 macroblock)
14
Katto lab.
Overlapped Motion Compensation
Block MC:Copy a macroblock in the previous frame to build a prediction frame.
Overlapped MC:Overlap and add multiple macroblocks in the previous frame with weights.
M.Ohta and S.Nogaki: “Hybrid Picture Coding with Wavelet Transform and Overlapped Motion Compensated Interframe Prediction Coding", IEEE Trans. SP, Dec.1993.H.Watanabe and S.Singhal: “Windowed Motion Compensation", SPIE VCIP-91, Nov.1991.
∑∑ −−=i nm
yixiii dydxznmwyxz,
,, ),(),(),(ˆ
),(),(ˆ yx dydxzyxz −−=
15
Katto lab.
Problems (at that time)
What is the optimum weighting coefficient for overlapped motion compensation ?
“Raised cosine” and “trapezoid” window functions were heuristically tried. Only one work (below) exists.
Can we theoretically compare overlapped motion compensation with fractional-pel (e.g. half-pel) motion compensation ?
At that time, MPEG-1 adopted half-pel motion compensation. Later, overlapped motion compensation was adopted by H.263.
M.T.Orchard and G.J.Sullivan: “Overlapped Block Motion Compensation: An Estimation-Theoretic Approach”, IEEE Trans. IP, Sep.1994.
16
Katto lab.
Problem Formulation
minimize
under
[ ]2)ˆ( zzE −
⎪⎪⎩
⎪⎪⎨
⎧
=
=
∑
∑−
=
−
=
1
ˆ1
0
1
0N
kk
N
kkk
w
zwzN : # of macroblocks to beoverlappedzk : k-th mackroblock
Wiener-Hopf equation
bw 1−= AoptA : correlation matrixb : correlation vector
17
Katto lab.
Optimum WindowsTheoretical: AR(1) model Experimental
Trapezoid Raised Cosine Proposed
Salesman +0.90 +0.90 +0.92Blue Jacket +0.42 +0.41 +0.46
Simulations
18
Katto lab.
Motion Model (1)
corresponding pixels (between current and previous) do not change from frame to frame (though previous pixel may not be on sampling grid)
spatial correlation between pixels decreases exponentially
Assumptions:
),(),( ,0,0,0,00 yyxx ddyddxzyxz Δ−−Δ−−=
[ ][ ]
[ ]0
20
,0,000
),(),(),( dEyx
yxzEdydxzyxzE Δ=
Δ−Δ−ρ
19
Katto lab.
Motion Model (2)Probability density function of motion estimation errors
[ ] max,0,0 21
41 VsdEa ii ⋅
−+=Δ=
[ ] ⎟⎠⎞
⎜⎝⎛ −⋅
−+=Δ=
43
21
81
max,0,0 VsdEa hh
reliableunreliable
[ ][ ]2
20 )(
zEzzE − (prediction error reduction)
Interger-pel ia ,01 ρ−
Half-pel ha ,01 ρ−
Overlapped 22/)(
2/)( 21
111
,1,0,1,0
,1,0
,0 ⎥⎦
⎤⎢⎣
⎡ +−−−
−−+
+
iiii
ii
i
aaaa
aaa ρρρ
ρρ
Half-pel + Overlapped 22/)(
2/)( 21
111
,1,0,1,0
,1,0
,0 ⎥⎦
⎤⎢⎣
⎡ +−−−
−−+
+
hhhh
hh
h
aaaa
aaa ρρρ
ρρ
overlapping effect
20
Katto lab.
Results
Overlapped motion compensation was originally proposed to reduce blocking artifacts in block motion compensation.Proposed theory validates the prediction error reduction effect of overlapped motion compensation. The theory proves that overlapped motion compensation performs well when motion estimation does not work well, though half-pelmotion compensation works well when motion estimation works well. They can be combined and can help each other.
21
Katto lab.
3. “Overlapped” Interframe Prediction
IPB prediction structureI-picture: Intraframe codingP-picture: Interframe codingB-picture: Bi-directional interframecoding
Adopted in MPEG-1 standard
J.Katto and M.Ohta: "Mathematical Analysis of MPEG Compression Capability and Its Application to Rate Control", IEEE ICIP'95 Oct.1995.
22
Katto lab.
Problems (at that time)
Why does the IPB prediction structure work better than the IP prediction structure ?
MPEG-1 was indeed better than H.261. One reason is half-pel motion compensation, and another is IPB prediction. But, why ?
TM5 (famous rate control for MPEG-2 standard at that time) is the best rate control method ?
Theory on IPB prediction might provide better rate control strategy.
23
Katto lab.
Coding Gain of IPB Prediction (1)
Prediction error variancesP-picture
B-picture( ) 22 )(12 xP Mcor σσ ⋅−⋅=
22 )()()(21
23
xB jMcorjcorMcor σσ ⋅⎟⎠⎞
⎜⎝⎛ −−−⋅+=
j M-j
cor(k) : correlation between k-frame apart frames
24
Katto lab.
Coding Gain of IPB Prediction (2)
( ))(12/)( 22 McorMP xP −⋅== σσ
⎟⎠⎞
⎜⎝⎛ −−−⋅+== )()()(
21
23/)( 22 jMcorjcorMcorMB xB σσ
IPB prediction structure
optimum bit allocationproblem
MM
LML
IPB
MMBMP
G 11
1)()(
1−
−
⎥⎦⎤
⎢⎣⎡
−⋅
=
wkkcor ρ=)(
25
Katto lab.
Rate Control for IPB Prediction (1)
HeuristicsbQaR +⋅= loglog
R : rateQ : quantization step sizea, b : constants
XRQ =⋅ α
Rate-distortion theory: 2222 2 xR
q σεσ −⋅=
difficult to apply to rate control directly ...
26
Katto lab.
Rate Control for IPB Prediction (2)
Optimum target bit assignment minimize average quantization step size
under rate constraint
BPI
mBB
mPP
mII
NNNQNQNQN
++⋅+⋅+⋅
constRNRNRN BBPPII =⋅+⋅+⋅
NI, NP, NB : # of pictures in GOP
27
Katto lab.
Rate Control for IPB Prediction (3)
Solution
αα mm
I
BB
mm
I
PPI
I
XXN
XXNN
RR++
⎟⎟⎠
⎞⎜⎜⎝
⎛+⎟⎟
⎠
⎞⎜⎜⎝
⎛+
=11
αmm
P
BBP
P
XXNN
RR+
⎟⎟⎠
⎞⎜⎜⎝
⎛+
=1
αmm
B
PPB
B
XXNN
RR+
⎟⎟⎠
⎞⎜⎜⎝
⎛+
=1
III XRQ =⋅ α
PPP XRQ =⋅ α
BBB XRQ =⋅ α
observable
Simulation
28
Katto lab.
Summary of Formulations
29
Katto lab.
Results
Coding gain of IPB prediction quantitatively proves its efficiency against IP prediction.
(However, though omitted, coding gain based rate control is not practical)
Rate control for IPB prediction based on heuristics between rate and quantization step size provides smarter bit assignment than TM5.
TM5 is included as a special case of the proposal.
30
Katto lab.
4. Denoising by “Overlapping”
Reconsiderations on video codingComplexity shift to decoders
Distributed video codingApplication to wireless sensor networks
Multiple description codingApplication to heterogeneous networks
Super-resolutionSDTV/HDTV conversion at a decoder
Image restorationDenoising
Reduction of blocking artifacts and flicker noisesDeblocking filters, motion compensated temporal filtering, re-application of JPEG, and so on ...
J.Katto, J.Suzuki, S.Itagaki, S.Sakaida and K.Iguchi: “Denoising Intra-Coded Moving Pictures using Motion Estimation and Pixel Shift”, IEEE ICASSP 2008, Apr.2008..
31
Katto lab.
Problems
Quality enhancement of compressed video streams becomes important
Enormously rapid increase of compressed visual contents over storages and networksPeople prefer better quality contents if they don’t pay much money for them Decoders (PCs) become powerfulSmart quality improvement of old visual contents is impressive Smart denoising might contribute to smarter compression in future
32
Katto lab.
Auxiliary Experiment
encode &decode
encode &decode
encode &decode
pixel shift(0,0)∼(7,7)
+
originalimage
reconstructedimage
encode &decode
encode &decode
encode &decode
pixel shift(0,0)∼(7,7)pixel shift
(0,0)∼(7,7)
+
originalimage
reconstructedimage
Pixel shift, JPEG compression and picture “overlapping” brings PSNR improvement.
0
0.5
1
1.5
2
2.5
3
1 10 100number of aligned images
PS
NR
gai
n [d
B]
Airplane (JPEG)Airplane (JPEG-2000)Whale (JPEG)Whale (JPEG-2000)theory (a=0.8)theory (a=0.6)theory (a=0.4)theory (a=0.2)theory (a=0.0)
a=0.6
33
Katto lab.
Formulation (1)
Statistical reduction of quantization errors
minimize
where
under constraint
∑∑==
+==K
k
K
kkqkwxkxkwx
11)()()(ˆ)(~
1)(1
=∑=
K
kkw
( )[ ]⎥⎥⎦
⎤
⎢⎢⎣
⎡⎟⎠
⎞⎜⎝
⎛=− ∑
=
2
1
2 )()(~ K
kkqkwExxE
K : # of overlapped picturesq(k) : k-th quantization errorw(k) : k-th weight
34
Katto lab.
Formulation (2)
Solutionvariant of Wiener-Hopf equation
where , and
uw 1−⋅= RCopt
t)1,,1,1( L=u)(sum
11 u−=
RC
[ ]⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡
=⋅=
])([)]2()([)]1()([
)]()2([])2([)]1()2([)]()1([)]2()1([])1([
2
2
2
KqEqKqEqKqE
KqqEqEqqEKqqEqqEqE
ER t
OM
L
correlation matrix of quantization errors
35
Katto lab.
Formulation (3)
Special caseswhen independent, i.e.
when completely dependent, i.e.no gain (corresponding to no pixel shift)
[ ] )(0)()( lklqkqE ≠=
∑=
⋅= K
l lqkq
opt kw
12
)(
2)(
11)(
σσ
∑=
= K
k kq
q
12
)(
2min, 1
1
σ
σ
Kkwopt
1)( =
Kq
q
22
min,
σσ =
22)( qkq σσ =esp. when
[ ] 2)()( qlqkqE σ=
)(log10 10 dBKGain:
36
Katto lab.
Motion JPEG Extension
pixelshift
MJPEGencode
MJPEGdecode
inverse pixel shift
motionestimation
alignment& mixture
encoder decoder
Constraint by quantization step sizes
reference frames
Motion estimation at a decoderIntentional pixel shift at an encoder and decoder (though breaking compatibility)
37
Katto lab.
Experiments (1)
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
fwdME biME shift+fwdME shift+biMEMotion Estimation
PSN
R g
ain
[dB]
akiyo (34.7dB)children (30.2dB)coastguard (29.2dB)container (28.7dB)flower (26.8dB)mobile (26.0dB)news (32.3dB)sean (30.2dB)weather (26.4dB)
motion shift and pixel shift cooperate
motion estimation effect
pixel shift effectfwd: forwardbi: bi-directionalshift: pixel shift
Motion estimation works well for moving regionsIntentional pixel shift works well for static regions
38
Katto lab.
Experiments (2)
Subjective quality comparisonMotion JPEG Proposed method
impressive reduction of blocking artifacts and flicker noises
39
Katto lab.
Results
Overlapping similar regions with different quantization noises contributes to statistical reduction of quantization noises.Subjective quality improvement in blocking artifacts and flicker noises is also observed.
Extension to interframe coding is ongoing.Extension to super-resolution is also ongoing.
40
Katto lab.
Conclusions
“ Overlapping” had brought many performance improvements in video compression.“Optimum bit allocation” is one of important key formulations.“Wiener-Hopf equation” is another important key formulation.