recent advances in video coding.thomas wiegand 2011
TRANSCRIPT
Thomas Wiegand
Heiko Schwarz
� Video Coding – State-of-the-Art
� High-Efficiency Video Coding• Motion representation• Motion compensation
• Transform coding• Intra coding
• Entropy Coding
slid
e 3
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
“It has been customary in the past to transmit successive
complete images of the transmitted picture.”
[...]
“In accordance with this invention, this difficulty is avoided by
transmitting only the difference between successive images of
the object.”
slid
e 4
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
1989: Digital TV – Digital Broadcast, DVD
1999: TML-1 of H.264/AVC
Today: more than 1 Billion devices with H.264/AVC
Every HDTV Receiver
Every Blu-Ray Player
Most Internet Video
Countless Mobile Video
slid
e 5
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
Source: http://techcrunch.com/2010/05/01/h-264-66-percent-web-video/
slid
e 6
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
EntropyCoding
Scaling & Inv. Transform
Motion-Compensation
ControlData
Quant.Transf. coeffs
MotionData
Intra/Inter
CoderControl
Decoder
MotionEstimation
Transform/Scal./Quant.-
InputVideoSignal
Split intoMacroblocks16x16 pixels
Intra-frame Prediction
De-blockingFilter
OutputVideoSignal
slid
e 7
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
Efficiency of Motion CompensationEfficiency of Motion Compensation• Intraframe coding: only spatial correlation exploited
� DCT [Ahmed, Natarajan, Rao 1974], JPEG [1992]
• Motion compensation (MC): integer-pel accuracy, 16x16� H.261 [1991]
• Half-pel accurate MC� MPEG-1 [1993], MPEG-2/H.262 [1994]
• 16x16 & 8x8 block size MC, quarter-pel� H.263 [1996], MPEG-4 [1999]
• Quarter-pel & 16x16 - 4x4 block size MC and multiple reference pictures
� H.264/AVC [2003]
• 64x64 – 4x4 block sizes and new filters for MC� HEVC [start 2010]
Complexityincrease
slid
e 8
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
Development of Video CodingDevelopment of Video Coding
0 100 200 300
28
30
32
34
36
38
40
Rate [kbit/s]
PSNR [dB] Half-pel
motion compensation(MPEG-1 1993MPEG-2 1994)
Integer-pelmotioncompensation(H.261, 1991)
Variable block size(16x16 – 8x8)(H.263, 1996) +quarter-pelmotion compensation(MPEG-4, 1998)
Variable block size(16x16 – 4x4) +quarter-pel +multi-framemotion compensation(H.264/AVC, 2003)
Intra frameDCT coding(JPEG, 1990)
Bit-rate Reduction: 75%
35
Foreman10 Hz, QCIF100 frames
slid
e 9
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
Development of Video CodingDevelopment of Video Coding
0 100 200 300
28
30
32
34
36
38
40
Rate [kbit/s]
PSNR [dB] Half-pel
motion compensation(MPEG-1 1993MPEG-2 1994)
Integer-pelmotioncompensation(H.261, 1991)
Variable block size(16x16 – 8x8)(H.263, 1996) +quarter-pelmotion compensation(MPEG-4, 1998)
Variable block size(16x16 – 4x4) +quarter-pel +multi-framemotion compensation(H.264/AVC, 2003)
Intra frameDCT coding(JPEG, 1990)
Foreman10 Hz, QCIF100 framesVisual Comparison
slid
e 10
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
Foreman, QCIF, 10 Hz, 100 kbit/sJPEG H.264/AVC
slid
e 11
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
� Video encoder needs to decide between many options denoted with parameter vector p
� Unconstrained Lagrangian Formulation:
min
pD(p) + λ ⋅R(p)
D - Distortion
R - Rate
RT - Target Rate
p - Parameter Vector
with λ controlling the rate-distortion trade-off
min
pD(p) s.t. R(p) ≤ RT
� Minimization tests the various modes in video coding[Wiegand, et al., 1996]
� Constrained Problem:
slid
e 12
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
[Shoham & Gersho, 1989]
slid
e 13
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
Next Steps in Video CodingNext Steps in Video Coding
0 100 200 300
28
30
32
34
36
38
40
Rate [kbit/s]
PSNR [dB] H.264/AVC (2003)?
slid
e 15
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
� New Joint Collaborative Team on Video Coding (JCT-VC)of ISO/IEC MPEG and ITU-T VCEG
� April 2010: Call for Proposals evaluated by JCT-VC• 27 proposals received• Significant video quality improvements relative to
H.264/AVC High Profile for high and low delay• JCT-VC decided to create Test Model under Consideration (TMuC)
� October 2010: Creation of first Test Model
� Early 2013: intended finalization of standard
slid
e 16
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
EntropyCoding
Scaling & Inv. Transform
Motion-Compensation
ControlData
Quant.Transf. coeffs
MotionData
Intra/Inter
CoderControl
Decoder
MotionEstimation
Transform/Scal./Quant.-
InputVideoSignal
Split intoMacroblocks16x16 pixels
Intra-frame Prediction
De-blockingFilter
OutputVideoSignal
8x8
0
4x8
0 10 1
2 3
4x48x4
1
08x8Types
0
16x16
0 1
8x16MB
Types
8x80 1
2 3
16x8
1
0
slid
e 17
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
� Division of a picture into square blocks
� Blocks are assigned to quadtrees
� Maximum block size is signalled (e.g. 64x64)
� Quadtree-based subdivision oftree block into prediction and transform blocks
� Rectangular shapes
� Geometric partitions
slid
e 18
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
CurrentPicture
4 Prior Decoded Picturesas Reference
∆ = 2
1. Extend motion vector by reference picture index ∆
2. Provide reference pictures at decoder side
Flexible buffering of reference pictures generalizes B pictures:• B pictures can be reference pictures (decoupling of concepts)• Reference picture for a B picture can be any prior decoded picture
∆ = 4
∆ = 1
3. In case of bi-predictive coding (B pictures): decode 2 sets of motion parameters
[Wiegand, et al., 1999]
slid
e 19
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
MotionMotion--Compensated InterpolationCompensated Interpolation
� Longer Interpolation Filters• In H.264/AVC: 6-tap filters + bi-linear
• In HM: 12-tap interpolation filters
� Generalized Interpolation• MOMS basis functions
• IIR pre-filtering of reference image (2-tap)
• Short FIR filters (4-taps)
� Adaptive Interpolation filters• Transmission of filter coefficients in slice header
• Transmission of an index into a list of predefined filters
� Directional interpolation filters• Low-complexity interpolation
slid
e 20
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
Motion Representation
� Quadtree-based splitting of blocks
� Motion Partition Merging
L
T
X
B
R
slid
e 21
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
Motion Vector Coding
� Adaptive Motion Vector Coding• Signaling of motion vector predictor
current block
toptop
righttopleft
left
bottomleft
slid
e 22
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
EntropyCoding
Scaling & Inv. Transform
Motion-Compensation
ControlData
Quant.Transf. coeffs
MotionData
Intra/Inter
CoderControl
Decoder
MotionEstimation
Transform/Scal./Quant.-
InputVideoSignal
Split intoMacroblocks16x16 pixels
Intra-frame Prediction
De-blockingFilter
OutputVideoSignal
Transform CodingTransform Coding
slid
e 23
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
1. Partitioning into prediction blocks (intra or inter prediction)
2. Partitioning of prediction blocks into transform blocks(specifying the transform sizes)
Partitioning of a „tree block“ into prediction blocks
Partitioning of a prediction block into transform blocks
slid
e 24
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
Variable Transform Block Sizes Variable Transform Block Sizes � Transform sizes range from 4x4 to 32x32
� Requirement on fast algorithms
� Additional new directional transforms proposed
slid
e 25
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
EntropyCoding
Scaling & Inv. Transform
Motion-Compensation
MotionData
Intra/Inter
CoderControl
Decoder
MotionEstimation
Transform/Scal./Quant.-
InputVideoSignal
Split intoMacroblocks16x16 pixels
Intra-frame Prediction
De-blockingFilter
OutputVideoSignal
Intra PredictionIntra Prediction� H.264/AVC directional prediction
� New directional predictions and transforms
1
2
3456
7
8
0Q A B C D E F G HI a b c dJ e f g hK i j k lL m n o p
slid
e 26
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
EntropyCoding
Scaling & Inv. Transform
Motion-Compensation
ControlData
Quant.Transf. coeffs
MotionData
Intra/Inter
CoderControl
Decoder
MotionEstimation
Transform/Scal./Quant.-
InputVideoSignal
Split intoMacroblocks16x16 pixels
Intra-frame Prediction
De-blockingFilter
OutputVideoSignal
Entropy CodingEntropy Coding
slid
e 27
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
� Entropy coding: reversible mapping from one data representation to another, more compact representation
• Entropy is lower bound on transmission rate [Shannon, 1948]
� Problem complicated in video coding
� Video encoder generates sequence of symbols
� Sequence of symbols contains a mix of data types: control data, motion vectors, transform coefficients
slid
e 28
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
� Symbols in video codec correspond to a mix of data type: control data, motion vectors, transform coefficients
� Huffman coding: create a code for each data type
� Arithmetic coding: binarize data and arithmetically encode each binary symbol
� Examples1,s2 ,...,sN{ }→ b1,b2,b3 ,b4 ,...,bB{ }
Symbol Binary Symbol
0 0
1 10
2 11
slid
e 29
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
� Simple construction method for entropy codes [Huffman, 1951]
� Minimizes average codeword length
with constraint
� Bounds on Rate
� Example
Redundancy 17.8 %
slid
e 30
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
� Alternative construction method for entropy codes [Rissanen, 1976], [Pasco, 1976]
� Achieves entropy bound asymptotically for
� In video coding: context-adaptive binary arithmetic coding(CABAC), [Marpe, Schwarz, Wiegand, 2003]
Context Modeling
Binarization Probability Estimation
Coding Engine
update probability estimation
Adaptive binary arithmetic coderBottleneck !
slid
e 31
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
� Arithmetic coding represents probability of binary symbols
� New concept: PIPE coding [Marpe, Schwarz, Wiegand 2010]
slid
e 32
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
0,0
0,2
0,4
0,6
0,8
1,0
0 0,1 0,2 0,3 0,4 0,5probability p
exp
ecte
d ra
te p
er b
in [
bit
]
entropy limitapproximation by
interval discretization
probabilityinterval
intervalrepresentative
� Uniform probability distribution of binary probabilities
� With slope of binary entropy function
� Rate increase
Average Rate Increase (relative to entropy limit)
f( p ) K=1 K=2 K=4 K=8 K=12 K=16
uniform 12.47% 3.67% 1.01% 0.27% 0.12% 0.07%
slid
e 33
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
� Assume fixed probability for each interval
� Create combined events of binary symbols
� Probability of symbol p
� Leaf node probability is
bin sequence codeword 000 11
001 001
01 01
10 10
11 000
0.216
0.144
0.240
0.240
0.160
11
001
01
10
000p
p2
qp
pq
qq2
q3
q2p
p = 0.4q = 1− p
prob. codewords
slid
e 34
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
� Coding efficiency of arithmetic coding
� Complexity of Huffman coding
0.000
0.001
0.002
0.003
0.004
0.005
0.006
0 0.1 0.2 0.3 0.4 0.5
probability p
aver
age
cod
ewo
rd le
ng
th in
crea
se [
bit
s]
optimal code assumption (theoretical)
real V2V codes (with up to 65 codewords)
0.12 %
0.24 %
slid
e 35
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
EntropyCoding
Scaling & Inv. Transform
Motion-Compensation
ControlData
Quant.Transf. coeffs
MotionData
Intra/Inter
CoderControl
Decoder
MotionEstimation
Transform/Scal./Quant.-
InputVideoSignal
Split intoMacroblocks16x16 pixels
Intra-frame Prediction
De-blockingFilter
OutputVideoSignal
InIn--Loop FilteringLoop Filtering
slid
e 36
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
InIn--Loop FilteringLoop Filtering
� Deblocking filter
� Adaptive Loop Filter• Wiener filter• Transmission of filter coefficients and indication for regions
slid
e 37
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
Coding EfficiencyCoding Efficiency
slid
e 38
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
� On average, current gains seem to be around 30% bit rate reduction
� Goal: 50% bit rate reduction against H.264/AVC HP at same quality
� Finalization: January 2013
Coding EfficiencyCoding Efficiency
slid
e 39
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
� H.264/AVC: everywhere today
� High-Efficiency Video Coding
• New project of MPEG and VCEG
• Started in April 2010
• First Test Model in Oct. 2010
• Currently roughly 30% bit rate reduction
• Goal: 50% bit rate reduction vs. H.264/AVC HP
• Finalization intended for early 2013
slid
e 40
Thomas Thomas WiegandWiegand: Recent Advances in Video Coding: Recent Advances in Video Coding
HHI & TUB members and research associates� D. Marpe� H. Schwarz� K. Müller� P. Kauff� R. Schäfer� other members of the HHI/TUB team