1 h.264 fast mode decision multimedia systems team project 林展意 695410074 林志仁 695410063
TRANSCRIPT
1
H.264 Fast Mode Decision
Multimedia Systems Team Project
林展意 695410074林志仁 695410063
2
• H.264 intra-prediction and inter-prediction– “Fast Macro-block Mode Determination to Reduce
H.264 Complexity”
– Fast Inter-mode Decision in H.264/AVC Video Coding
– “Feature-Based Intra-Prediction Mode Precision for H.264”
• Fast intra 4*4 block mode decision
– “Fast Mode Decision for H.264”• Fast P frame encode
3
Outline
Introduction
Fast mode decision
Comparison and Conclusion
Reference
Frame Encoding
H.264 Frame
4
Outline
Introduction
Fast mode decision
Comparison and Conclusion
Reference
H.264 Frame
Frame Encoding
5
H.264 has Certain distinct features such as
variable block size Motion Estimation, multiple
reference frames, and so on.
Introduction
H.264 uses rate distortion optimization technique to
get the best coding result in terms of maximizing
coding quality and minimizing resulting data bits.
6
7
However, the complexity of the codec and the
computational cost increase tremendously, how to
reduce the complexity of the algorithm is a
valuable problem here.
Introduction
8
Outline
Introduction
Fast mode decision
Comparison and Conclusion
Reference
Frame Encoding
H.264 Frame
I - FrameP - Frame
9
I - Frame
P - Frame
Frame EncodingIntra Frame
For Every
Macro-Block
For every 8*8 block
DCT
Quantization
Entropy Coding
110010110….
YCb
Cr
RGB
• Macro-block– 16*16 RGB macro-block
for YCbCr transformation– 4:2:0 chromance format – Block size 8*8 for DCT
transformation– Fixed quantization step
size– Context free entropy
coding• Huffman coding
10
I - Frame
P - Frame
Frame Encoding
• Motion vector Block matching algorithm
• Sequential search
• Binary search
11
I - Frame
P - Frame
Frame Encoding
Target frame
Reference frame
16
16
MV
Current macro-block
Best match- Y
Cb
Cr
Difference macro-block
For every 8*8 block
DCTQuantization
Entropy Coding
110010110….
This process of finding the best match is known as Motion Estimation
12
Outline
Introduction
Fast mode decision
Comparison and Conclusion
Reference
Frame Encoding
H.264 Frame
Intra-prediction Inter-prediction
13
16 168 8
81616
8
888
8
44
44
The method of partition macro-blocks into motion compensated sub-blocks of varying size is known as tree structured motion compensation.
14Block sizes chosen after H.264 RDO. Ex. CIF sequence “Paris”
15
Fast mode decision for H.264
• P frame encode– “Fast mode decision for H.264”
• Decision skip mode or inter-prediction or intra-prediction– 1. Inter-prediction
• Temporal
• Skip mode , or block size decision
– 2. Intra-prediction• Spatial
• H.264 can do it
• but other codec using JPEG-like encode
16
High computation need
降低計算量且維持 H.264 quality : fast mode decision • Macro-block 16*16block size decison
• Intra-prediction : mode decision to find optimal residue matrix and block size decision
– 16*16 or 4*4
– Intra-prediction optimal SAD(SATD) cost
– Intra-prediction optimal RD cost
• Inter-prediction : optimal motion vector decision to find residue matrix and block size decision
– 16*16,16*8,8*16,8*8,8*4,4*8,4*4
– Using residue matrix
– Inter-prediction optimal SAD cost
– Inter-prediction optimal RD cost
Inter RD cost
Intra RD cost
Inter SATD
Inter SAD
Intra SAD Intra SATD
17
Intra-prediction prediction:
spatial redundancy • Encoder decide intra-
prediction– 1. 16*16 and 4*4 mode
decision• SAD or SATD cost• RD cost
– 2 . decide block size• Block size
– 16*16 or 16 個 4*4• Mode decision
– 16*16 have 4 mode– 4*4 have 9 mode decision– (fast 4*4mode decision)
• “feature-based intra-prediction mode decision for H.264”
18
Inter-prediction mode
• 1. find combination block size optimal motion vector and residue matrix
• Residue matrix to calculate– SAD , low complexity to
calculate quality– Or RD cost, high
complexity to calculate quality
• 2. decide block size combination
19
How to find the best matching block ?
Mean Square Error (MSE)
Mean Absolute Error (MAE)
Sum of absolute Difference (SAD)
Rate Distortion Optimization (RDO)
s=1, MAE ; s=2, MSE
1M N
M ¡ 1P
k=0
N ¡ 1P
l=0jC(x +k;y+ l) ¡ R(x + i +k;y+ j + l)js
M ¡ 1P
k=0
N ¡ 1P
l=0j(C(x + k;y+ l) ¡ R(x + i + k;y+ j + l))sj
20
Outline
Introduction
Fast mode decision
Comparison and Conclusion
Reference
Frame Encoding
H.264 Frame
21
Fast Macro-block Mode Determination to Reduce H.264 Complexity
Ki-Hun HAN, Yung-Lyul LEE
IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences Issue 3 (March 2005) 800-804
OXFORD Journal
22
Using the RDO technology to decision the macro-block mode in the H.264 encoder.
The proposed method reduces the encoder complexity by 55% and reduces approximately 10 % of bit-rate.
The RDO mechanism is used to select the best MB mode among all of the blocks mode.
Summary
23
Rate Distortion Optimization (RDO)
RDcost : To choose the best MB mode that has the minimum rate distortion value.
Distortion = MSE
¸mode
RDcost = Distortion+ ¸modeRates
, the Lagrangian parameter , is a function related to the H.264 quantization parameter (QP), whose value ranges from 0 to 51.=
0:85¢2(Q P ¡ 12)
3
24
P-Frame
Inter mode decision
Intra mode decision
Skip
Inter 16*16
Inter 16*8
Inter 8*16
Inter 8*8
Intra 16*16
Intra 4*4
Inter 8*8
Inter 8*4
Inter 4*8
Inter 4*4
P-Frame mode decision
25
P-Frame mode decision
If the RDcost value of the Inter-16*16 mode is the smallest of the three mode, it is preferable to select the Inter-16*16 MB mode.
For each Macro-block
ME of Inter-16*16, Inter16*8, Inter 8*16 modeME & RDcost computation
If RDcost of Inter-16*16 mode
is minimum among 3 modes
P8*8 mode ME & RDcost computation
Best Inter mode decision
Inter-8*8 SkipNo
Yes
26
P-Frame mode decision
Intra mode decision
Vertical Spatial Prediction Errors (VSPE)
Horizontal Spatial Prediction Errors (HSPE)
27
P-Frame mode decision
Intra mode decision
Intra-16*16If ((SPE > TPE) and (SPE >= 20))
Intra-16*16 mode RDcost computation
Intra-16*16 Skip
No
Yes
TPE is the sum of the SAD between the current MB and predicted MB with respect to the best Inter MB mode and the motion vector bits.
28
P-Frame mode decision
Intra mode decision
Intra-4*4
Intra_RD = (Sum of the RDcost of all MBs encoded in the Intra-4*4 mode that has computed until now) / n
If Best_Inter_mode_RDcost <
Intra_RD
Intra-4*4 mode RDcost computation
Intra-4*4 Skip
No
Yes
29
30
31
Intra_RD = CurRD+n¢Intra RDn+1
CurRD is the RDcost value when the current MB is selected as the Intra 4*4 mode.
n denotes the number of intra 4*4 MBs.
32
Result
Recommended common experimental conditions by H.264 standard group
33
Experimental results
ETime : total reduction in computation timeResult
34
Fast Intermode Decision in H.264/AVC Video Coding
D. Wu, F. Pan, K. P. Lim, S. Wu, Z. G. Li, X. Lin, S. Rahardja, and C. C. Ko
IEEE Journal Circuits and Systems for Video Technology, IEEE Transactions onVolume 15, Issue 7, July 2005 Page(s):953 - 958
35
It makes use of the spatial homogeneity and the temporal stationarity characteristics of video objects.
This paper proposes a fast algorithm is able to reduce on the average 30% encoding time and a bit rate increment of 0.6%.
Fast Intermode Decision in H.264/AVC Video Coding
spatial homogeneity
temporal stationarity
36
Homogeneous blocks in the picture would have similar motion and are very seldom split into smaller blocks.
There is an average of 35% homogeneous in a typical video frame, and these areas are suitable for larger size inter-mode coding, and RD cost calculation of small size modes can be avoided.
Specifically, spatial homogeneity of a MB is decided based on the MB’s edge intensity.
Fast Intermode Decision in H.264/AVC Video Coding
37
(i, j)
(i-1, j+1)
(i, j+1)(i+1, j+1)
(i-1, j-1) (i, j-1)
(i, j+1)
dxi ;j =+ 2 + - - 2 - + 2 + - - 2 -
dyi ;j =
(i, j)(i-1, j)
(i+1, j)
(i+1, j-1)(i-1, j-1)
(i-1, j+1)
(i+1, j+1)
If the sum of magnitude of the edge vectors at all pixel locations in the block is less than the block homogeneous threshold, ThdH :
ThdH is set to 20,000 for 16*16 block and 5,000 for 8*8 block .
Homogeneous Regions Determination
38
Stationary Regions Determination
We can use the sum of absolute difference to check if this MB changes or not.
Di® =i=1;j =1P
16;16abs(M [i; j ]¡ N [i; j ])
If the change between the two MBs is less than a threshold, Ts, the MB is classified as stationary and 16*16 mode is used for motion estimation.
Based on the experimental results, we set the threshold Ts to 200.
39
Perform edge detection on one image
Generate the edge direction histogram.
Predict if the current
MB has zero motion
Compute the MB difference.
If the sum is greater than the threshold
Yes
No
No
Yes
40
Perform RDOon the
16*16 block
If the MB is homogeneous
Perform RDO on the 16*16 blockand selectively on 16*8 or
8*16 block
NoYes
Perform RDO on the 16*16, 16*8 and 8*16 blocks. Save the best
mode of the three
41
Perform RDO on the 8*8 blockAnd the best type is selected
to be 8*8
For each 8*8 block in theMB, if is homogeneous
No
Yes
Perform RDO on 8*8, 8*4, 4*8 and 4*4 blocks and choose the best block subtype
All 8*8 blockhave selected
best mode
No
Yes
42
Determine the best mode between P8*8 type and the
best type from P16*16
Select the ultimate best mode among the intra-mode,
inter-mode and Skip mode
43
Result
Average -30.27 -0.0315 0.65
44
IEEE Conference Proceeding Volume 2, 24-27 Oct. 2004 Page(s):769 - 772 Vol.2
Feature-based intra-prediction mode decision for H.264
Changsung Kim, Hsuan-Huei Shih,C. C. Jay Kuo
45
“feature-based intra-Prediction mode decision for H.264” outline
• Quality calculation – RDO ( RD optimization ) method must try to all coding modes at
every block– STD,STAD , RD quality calculation
• Quality calculation filtering heuristic– STADSTADRD
• Flow chart of the proposed method– Intra-prediction mode decision schme
• Experiment results– Simulation condition – Computation evaluation– Quality evaluation
46
Quality calculation filtering heuristic• Two method decide quality
– Low computation but quality evaluation is not always good
• Calculate SAD or SATD
– High computation but quality evaluation ,quality evaluation is good
• Calculate SSD
• I(x,y): 原本的 pixel 值,• P(x,y): predict 的 pixel 值• D(x,y) :difference 的值• T{D(x,y)} : 對 D(x,y) 做 Hadamard transformation• SAD: summation of absolute difference• SATD: summation of transformation absolute
difference• Bias: 其他的 overhead• RD :rate distortion • SSD :the sum of the squared difference of s and c
– S : original signal– c: : reconstructed signal– 選擇這個 mode 的 overhead
• QP: quantization parameter– Decision by bit rate
Low complexity calculation
High complexity calculation
47
透過 SAD 還有 STAD 計算、利用 threshold 判斷是否再向下計算 RD cost 的 mode decision 的效果。利用設定 threshold 再和其比較,大則要算 RD cost 、反之即可 mode decision
Using (x,y)=(SAD,SATD)
statistically 93~95 % RDO mode fall in the of 3*3 window
48
Multi-stage 4*4 block mode decision降低計算量且維持品質的方法
• Filtering heuristic• Design early termination thresholds
– For hold quality not just calculate STAD
• 由觀察 SATD 比 STD 做 mode decision 更可能和 RDO 所做的 mode decision 相同、用此概念做 filtering ,降低計算量而且可以維持品質
• Threshold 值 T’th,Tth 由實驗值去決定
• 1. 先做 SATD 選出最小的二個 mode的 SATD 求出此 T’ , if T< T’th,mode decision else step2
• 2 再做 SAD 計算挑出 mode 最小的 2個 mode 求出 T, if T<Tth ,mode decision else step3
• 3. 由 SAD 還有 SATD 的計算找出2~3 mode 再去做 RDO 計算 做 mode decision
• RDO: rate distortion optimization
49
Simulation and calculation
• H.264 Jm7.3a codec • 以全部用 RD 做
mode decision 做比較– Quality 差異不大– 效能可提升 10~15 倍
50
IEEE Conference Proceeding Volume 2, 27-30 June 2004 Page(s):1131 - 1134 Vol.2
Fast mode decision for H. 264
Jeyun Lee, Byeungwoo Jeon
51
“Fast mode decision for H.264” outline
Flow chart of the proposed method in P frame encode– Skip mode decision at early stage in 16*16 block inter-
prediction– Heuristic method
• Selective intra mode decision• Decide to do intra-prediction or not
• Experiment results– Simulation condition – Performance evaluation
52
Selective intra mode decision(1)
• AE: average rate– The number of bits for
texture data without header– The average number of bits
consumed to encode the motion –compensated residual data under the best inter mode as an indicative of degree of temporal correlation
– Decide best inter-prediction– Calculate RD cost and
calculate AR
For Every
Macro-Block
Using 4:2:0 sub-sampling
YCb
8*8 block
53
Selective intra mode decision(2)
• Intra-prediction uses the pixels of already encoded neighboring block
• Boundary 和 block 邊緣的值相差不大則用此做預測效果好– 可以有較小的 residue 值
• ABE: average boundary error• 1 個 116*16 的 luma block 和二個 8*8 Cb,Cr block• ABE<AR 決定做 intra-predicton 且做出的結果和 inter-predicon 最
好的結果比
Case4 Left: available , Upper : available
54
Proposed algorithm
• 1.Inter Skip mode decision at early stage
– 先找出 16*16 的 block 是否可設為 skip mode
– Reference frame is just one previous one
– Transform coefficients are all quantized to zero
– 如果成立可以忽略其他的block size RD cost 計算
• 2. 計算 inter 下的 AR 值• 3. Selective intra mode decision
– Spatial – AR :average rate– ABE: average boundary– Omit all intra modes if AR<ABE– Else find out the best intra mode
and compares it with that of the best inter mode
55
Experiment: performance and quality• Computing RD cost 降 72%• Encoding time 降 30%• Simulation condition • S: the percent saving in number
of computing RD cost• • means the percent processing
time difference• BD-Bit rate• BD-PSNR• Other paper metrtic
56
Outline
Introduction
Fast mode decision
Comparison and Conclusion
Reference
Frame Encoding
H.264 Frame
57
Stefan (CIF)
Foreman (QCIF)
News
(QCIF)
Slient
(QCIF)
Container
(QCIF)
Paris (CIF)
58
PSNR(dB)
[1]
[2] [3] [4]
Sequence
Foreman (QCIF) -0.06 -0.062 -0.13 -0.065
News (QCIF) -0.04 -0.065 0.000
Container (QCIF)
-0.3 -0.012 -0.067
Slient (QCIF) -0.04 -0.022 -0.028
Paris (CIF) -0.04 -0.040 -0.048
Mobile (CIF) -0.05 -0.005 -0.41 -0.003
Average -0.08 -0.0343 -0.27 -0.0351
Fast Intermode Decision in H.264/AVC Video Coding
Fast Macro-block Mode Determination to Reduce H.264 Complexity
Feature-based intra-prediction mode decision for H.264
Fast mode decision for H. 264
59
Time(%)
[1]
[2] [3] [4]
Sequence
Foreman (QCIF) -48.82 -25.18 -11.316 -20.24
News (QCIF) -59.36 -42.62 -36.11
Container (QCIF)
-33.95 -36.25 -36.48
Slient (QCIF) -58.49 -45.16 -33.75
Paris (CIF) -63.03 -31.90 -34.69
Mobile (CIF) -60.76 -9.97 -13.805 -24.11
Average -54.06 -30.27 -12.56 -30.89
Fast Intermode Decision in H.264/AVC Video Coding
Fast Macro-block Mode Determination to Reduce H.264 Complexity
Feature-based intra-prediction mode decision for H.264
Fast mode decision for H. 264
60
Outline
Introduction
Fast mode decision
Comparison and Conclusion
Reference
Frame Encoding
H.264 Frame
61
Main Paper Reference
Chang, A. Wong, P.H.W. Yeung, Y.M. Au, O.C. Fast multi-block selection for H.264 video coding IEEE 2004 III- 817-20 Vol.3
Dajun Wu, Feng Pan, Keng Pang Lim, Si Wu, Zhengguo Li, Xiao Lin, Susanto Rahardja, Chi Chung Ko Fast Intermode Decision in H.264/AVC Video Coding IEEE Journal Volume 15, Issue 7, July 2005 Page(s):953 - 958
62
Main Paper Reference
Jeyun Lee, Byeungwoo Jeon Fast mode decision for H. 264IEEE Conference Proceeding Volume 2, 27-30 June 2004 Page(s):1131 - 1134 Vol.2
Changsung Kim, Hsuan-Huei Shih,C. C. Jay Kuo Feature-based intra-prediction mode decision for H.264IEEE Conference Proceeding Volume 2, 24-27 Oct. 2004 Page(s):769 - 772 Vol.2
63
Paper Reference
Ki-Hun Han, Yung-Lyul Lee Fast Macro-block Mode Determination to Reduce H.264 Complexity IEICE Issue 3 (March 2005) 800-804
Wei, Zhenyu; Ngan, King NgiA Fast Macro-block Mode Decision Algorithm for H.264IEEE Conference Proceeding 4-7 Dec. 2006 Page(s):772 - 775
64
Iain E. G. Richardson “H.264 and MPEG-4 video compressionvideo coding for next-generation multimedia,” The Robert Gordon University, Aberdeen, UK, WILEY 2003
Book Reference
Li. Drew 原著 徐永煜‧莊舜清‧陳岳宏‧曾政龍‧賴柏伸‧鄭士賢‧謝宗儒‧呂宜玲‧沈揚智‧李卓育‧陳建榮‧蔡富評‧聶政邦 譯 傅心家 審閱譯 多媒體概論 (Fundamentals of Multimedia) 高立圖書有限公司
65
Report Reference
資工所 P76924194 蘇琬婷多媒體通訊期末報告 - 快速動作估計演算法 Fast Block Motion Estimation
曾紹崟 Shau-Yin Tseng 許志高 Ji-Gao Hsu多媒體壓縮標準 H.264 面面觀 多媒體應用技術部 SoC Technical Journal
郭其昌H.264 先進視訊編解碼標準工研院電通所
66
E N D
T hanks for y Uo r lis nte
67
Appendix