Download - H264 Review and Comparison
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 1/72
Review and Comparison
Francisco [email protected]
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 2/72
Introduction Video compression standards Video encoding/decoding
Fundamentals What is a video sequence?
Colour spaces▪ RGB
▪ YUV
Prediction▪ Temporal prediction▪ Spatial prediction
▪ Frame types
H.264/AVC H.264 components
Macroblocks Slices
Decoding/Display order
Reference picture lists
H.264 prediction Intra-prediction Inter-prediction
Transforms and quantization H.264 coding Summary of encoding The High Efficiency Video
Coding (HEVC) What the HEVC?
Main differences between H.264and H.265
Conclusions References
2
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 3/72
3
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 4/72
«Consumer internet video streaming accounted for 56exabytes of internet traffic in 2010, it’s predicted that
this will grow to 403 exabytes by 2015»- Cisco,2011
“It would take over 6 million years to watch the
amount of video that will cross global IP networkseach month in 2016. Every second, 1.2 millionminutes of video content will cross the network in2016”
-Cisco, 2012
4
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 5/72
Why are standards so important?
Simplify inter-operability between
encoders/decoders from different manufacturers Make possible to build complete platforms
Avoid patent issues
5
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 6/72
Evolution of video compression standards
1990 1993 1994 1995 - 1998 2003
H.261(VCEG)
Videoconferencing
MPEG-1 Part 2(MPEG)
VCD
Interactive CDsH.262/MPEG-2Video (JCT-VC)
DVD-VHDTV
H.263 (VCEG)H.263V2
MPEG-4 Part 2(JCT-VC)
Flash video
MPEG-4 Part 10H.264/AVC
(JCT-VC)
Mobile applicationsAdaptive multi-bitratestreaming
HDTV broadcastingBlu-ray Video3DTV
ITU-T Video Coding Experts Group (VCEG)
ISO/IEC Moving Picture Experts Group (MPEG)VCEG + MPEG = Joint Collaborative Team on Video Coding (JCT-VC)
Are we missingsomething?
6
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 7/72
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 8/72
We will refer to encoding as something«wider» than coding.
Encoded object
Coded info1
Coded info2
Coded infon
, … ,,
Mechanism
1Mechanism
2
Mechanism
n
8
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 9/72
We decode when we take «something» codedor encoded and recover its original «form».
9
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 10/72
What is a video sequence?
A video is a representation of a real-world visual scene
▪ The scene is sampled at a point in time to produce a frame
▪ Sampling is repeated at intervals of time to produce a motionpicture
The Hobbit: AnUnexpected Journey,was filmed and
released at 48 FPS!10
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 11/72
Digital video is just sampling a continuossignal spatially and temporally.
11
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 12/72
To obtain a 2D spacially sampled image(frame) usually CCD or CMOS sensors are
used to capture each color component
12
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 13/72
Digital video applications need a way tocapture and represent colour information
Two colour spaces will be explained▪ RGB
▪ YUV
13
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 14/72
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 15/72
Luminance (Y) and Chrominance (UV) canrepresent a color image
Luminance represents the achromatic image Chrominance represents the color information
15
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 16/72
The human visual system (HVS) is less sensitive tocolour than to luminance
Chrominance information can be stored with less
resolution than luminance
In the RGB colour space the three colours areequally important
YCrCb can be calculated from RGB values
Digital video uses Luma and Chroma (Y’CrCb)
= + + = −
= −
= − 16
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 17/72
Video sequences contain high levels of spatialand temporal redudancy
The goal of a prediction model is to reduce thisredudancy
Prediction from previously coded frames is calledtemporal (Inter)
Prediction from previously coded image samplesin the same frame is called spatial (Intra)
17
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 18/72
The predicted frame is created from one or morepast or future frames known as reference frames
Frame n, Frame n+1, Difference
Motion estimation helps to reduce the information to code
18
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 19/72
It is possible to estimate the trajectory ofeach pixel between successive video frames
Lighting changes and uncovered regions have tobe coded
Optical flow between frames
With a trajectory byeach pixel, this is stilltoo much information
19
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 20/72
Instead of calculating trajectories for eachpixel, blocks can be used. Take a MxN-sample region and search a similar
area en in the reference frame (smallest residual)– Motion estimation
Calculate the residual between regions – Motioncompensation
Code the residual block and the offset betweenthe current block and the position of the predictor(Motion vector)
20
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 21/72
Inter-prediction in block based video compression
standards21
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 22/72
Spatial prediction uses as reference frame thecurrent frame with its previously coded
blocks
Intra-prediction available samples
Extrapolation modesand residuals are coded
22
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 23/72
Depending on the used prediction, framescan be classified as:
Intra-prediction frames: Only spatial prediction isused, this frames are inserted to refresh the inter-prediction
Inter-prediction frames:
▪ Predictive (P)
▪ Bi-predictive (B)
23
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 24/72
Characteristics
Enhanced compression efficiency
▪
Gain of 50% of bitrate compared to previous standards Decoder complexity is increased by 4x with
respect to MPEG-2 decoder
Encoder is 8x more complex than a MPEG-2
encoder
Network friendly
24
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 25/72
25
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 26/72
An H.264 video encoder carries out prediction,transforming and coding processes to produce acompressed H.264 bitstream
Video codec: high level overview
26
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 27/72
The H.264 basic block is the Macroblock (MB)
Maximum sizes of 16x16, 8x8 and 4x4
Intra- and Inter-prediction are performed at MB level
A MB is formed byLuma and Chroma
blocks27
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 28/72
Each encoded frame is composed of one or moreslices
Slice header
Integral number of MB
Possible scenarios:
One slice per coded picture
N slices per picture (variable size) N slices per picture (fixed size)
Intra-perdiction is performedover the MBs contained in the
same slice 28
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 29/72
Decoding and Display order are not always thesame
Decoding order is related to the inter dependences
Display order depends on the original video sequence
29
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 30/72
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 31/72
Little example
31
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 32/72
Reference pictures are ordered in one or twolists prior to encoding or decoding a slice
0 1
For P slices, only 0 is used
Short term reference pictures are in decodingorder
For B slices, both 0 and 1 are used
Short term reference pictures are in display order
32
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 33/72
0 (P slice): The default order is decreasingorder of decoding
0 (B slice): The default order is:
Decreasing order of POC, for pictures with POCearlier than the current picture
Increasing order of POC, for pictures with POC later than the current picture
1 (B slice): The default order is: Increasing order of POC, for pictures with POC later
than the current picture
Decreasing order of POC, for pictures with POCearlier than the current picture
33
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 34/72
Little example for P slices (0)
34
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 35/72
Now with B slices (0, 1), frame 68
35
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 36/72
We can say that 0 is used to predict from“past” pictures while 1 is for “future”
pictures
Even when this is boring it isfundamental when we are trying
to achieve high compression
36
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 37/72
H.264/AVC supports a wide range ofprediction options
Intra-prediction▪ Many different prediction modes
Inter-prediction
▪ Motion compensation
▪ Sub-pixel interpolation
Multiple MB sizes (and luma block sizes)
Filtering process to reduce artifacts
37
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 38/72
Some facts:
An intra (I) MB is coded without referring to any
data outside the current slice I-MB may occur in any slice type
Every MB in an I slice is an I MB
Intra prediction uses samples from adjacent,previously coded blocks to predict the values inthe current block
38
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 39/72
Best block size selection is performed atencoder, there are different prediction modes
for each block sizeIntra-prediction block size Notes
16x16 (luma) Four possible predictionmodes
8x8 (luma) Nine possible predictionmodes
4x4 (luma) Nine possible predictionmodes
Chroma One prediction mode
39
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 40/72
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 41/72
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 42/72
Example of intra block size choices 42
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 43/72
Some facts: Inter-prediction blocks can range in size from 16x16
down to 4x4
Reference picture is chosen from DPB The offset between the current partition and the
prediction region in the reference picture is a motionvector (MV)
MV can point to integer, half or quarter-sample
positions in the luma component and even to aeighth-sample position for chroma
MVs are differencially coded from the MVs ofneighbouring blocks
43
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 44/72
More facts!:
A prediction block may be generated from asimple prediction region in a reference picture, fora P- or B-MB, or from two prediction regions inreference pictures, for a B-MB
Prediction block may be weighted according to
the temporal distance In B-MB, a block may be predicted in direct mode
44
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 45/72
An example of bi-prediction
45
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 46/72
Each 16x16 P- or B-MB may be predictedusing a range of block sizes
Each block or partition can have its own MV Partitions are not necessary square Motion vectors are coded using a predictive
approach
46
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 47/72
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 48/72
In H.264/AVC the transform and quantizationprocesses are designed to minimizecomputational complexity and avoid
ecoder/decoder mismatch. This is achievedby: Using an integer transform that can be carried out
using integer or fixed point arithmetic
Minimizing the number of multiplications,required by the quantization step to process ablock of residual data
48
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 49/72
Forward transform and quantization
Re-scaling and inverse transform
49
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 50/72
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 51/72
Re-scaling and inverse transform 51
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 52/72
A coded H.264 stream consists of a series ofcoded symbols.
The H.264/AVC standard specifies several
methods for coding symbols into a binarypattern: Fixed length code: A symbol is converted into a binary
code with a specified length (n bits).
Exponential-Golomb variable length code: Thesymbol is represented as a codeword with varyingnumber of bits (v bits). Shorter codewords areassigned to frequent symbols.
52
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 53/72
CAVLC: Context-Adaptive Variable LengthCoding, specially-designed method of codingtransform coefficients in which different sets of
VLC are chosen depending on the statistics ofrecently-coded coefficients.
CABAC: Context-Adaptive Binary Arithmetic
Coding, a method of arithmetic coding in whichthe probability models are updated based onprevious coding statistics.
53
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 54/72
Symbols occurring in the syntax above theslice data level (headers, video parameters)
Fixed Length Codes or Exp-Golomb codes. Symbols at slice data level and below
(Prediction modes, coefficients, etc.)
CABAC.
CAVLC + Fixed Length Codes or Exp-Golombcodes.
54
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 55/72
Typical H.264 encoder
55
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 56/72
Developed by the Joint Collaborative Team on Video Coding Designed to fulfill the requirements of the new video applications Focused on:
Increased video resolution
Parallel processing architectures It will allow:
Full-HDTV broadcasting
UHDTV (up to 8k*4k resolution) Mobile HD content
Full-HD 3DTV Provides an improvement of coding efficiency over AVC of 50%
(half bitrate same quality) HEVC encoding process is up to 10x more complex than the one
of AVC
56
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 57/72
H.265/ HEVC - MPEG-H Part 2 We will resume the encoding process into five
steps:
Partitioning
Prediction
Reconstruction
Coding
Packetization
57
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 58/72
HEVC introduces a largerblock structure thanprevious standards
A larger block structure
provides a highercompression performance
Basic block is known as thelargest coding unit (LCU), itcan be recursively split into
smaller coding units (CU) The CU is used as the basic
unit for intra- andintercoding
Picture partitioning
58
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 59/72
Picture partitioning/Prediction Units
59
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 60/72
A transform unit (TU) is thebasic unit for the transformand quantization processes
Size and shape of the TUsdepend on the size of the
PU TUs can be as small as 4x4
or as large as 32x32
Picture partitioning/Transform Units
60
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 61/72
Picture partitioning/Example
61
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 62/72
Prediction/Intra
HEVC has 35 luma and 6 chroma intraprediction modes(H.264/AVC has 9 luma and 4 chroma intra modes)
62
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 63/72
Asymmetric motion partitions (AMP) improve thecoding efficiency (allows to fit PUs to shapes in thepicture)
The accuracy of motion compensation in HEVC is1/4 pel for luma samples
Motion information is coded using advanced motionvector prediction (AMVP), merge and skip modesare availables
Prediction/Inter
63
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 64/72
HEVC applies square and non-square DCT-like integer transforms
Integer transforms used in HEVC are betterapproximations to the DCT than the used inH.264/AVC
Integer discrete sine transform (DST) is usedfor some residuals
Reconstruction/T&Q
64
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 65/72
HEVC applies 3 different in-loop filtering methods:
Deblocking filter: Similar to the one in H.264/AVC Sample Adaptive Offset (SAO): Classify pixels into
different categories and adds a simple offset to each pixelbased on its category
Adaptive Loop Filtering (ALF): Its constructed based onthe original image and it's designed to minimize thedistortion between the reconstructed and the originalimages
Reconstruction/In-loop Filtering
65
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 66/72
Coding/Tiles
66
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 67/72
Coding/Slices
67
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 68/72
Wavefront parallelprocessing (WPP) is anefficient mechanism forparallel encoding/decoding
Prediction dependenciesare not broken across slices
The basic concept is tostart processing a new rowof LCUs with a new parallelprocess as soon as twoLCUs have been processedin the row above
Coding/WPP
68
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 69/72
69
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 70/72
Even when HEVC is the new videocompression standard, it has a long path untilits complete adoption.
Most of the technologies that will need HEVCto work are still in an early stage.
H.264/AVC will continue to be the industry
standard for a while. A lot of research is being doing in different
HEVC related areas.
70
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 71/72
Ostermann, J., Bormans, J., List, P., Marpe, D., Narroschke, M.,Pereira, F., Stockhammer, T., et al. (2004). Video coding withH.264/AVC: tools, performance, and complexity. IEEE Circuits andSystems Magazine, 4(1), 7–28. doi:10.1109/MCAS.2004.1286980
Ohm, J., Sullivan, G. J., Schwarz, H., Tan, T. K., & Wiegand, T.
(2012). Comparison of the Coding Efficiency of Video CodingStandards – Including High Efficiency Video Coding (HEVC). Sullivan, G. J., Ohm, J., Han, W., & Wiegand, T. (2012). Overview of
the High Efficiency Video Coding (HEVC) Standard, (c). Bross, B., Han, W.-J., Ohm, J.-R., Sullivan, G. J., & Wiegand, T.
(2012). High Efficiency Video Coding (HEVC) Text Specification
Draft 8 (pp. 1–261).
71
8/12/2019 H264 Review and Comparison
http://slidepdf.com/reader/full/h264-review-and-comparison 72/72
Goldman, M. S. (2011). High Efficiency Video Coding (HEVC) — TheNext Generation Compression Technology. SMPTE Conferences,2011(1), 1–11. doi:10.5594/M001098
Pourazad, M., Doutre, C., Azimi, M., & Nasiopoulos, P. (2012).HEVC: The New Gold Standard for Video Compression: How Does
HEVC Compare with H.264/AVC? IEEE Consumer ElectronicsMagazine, 1(3), 36–46. doi:10.1109/MCE.2012.2192754 Nightingale, J., Wang, Q., & Grecos, C. (2012). HEVStream: a
framework for streaming and evaluation of high efficiency videocoding (HEVC) content in loss-prone networks. IEEE Transactionson Consumer Electronics, 58(2), 404–412.
doi:10.1109/TCE.2012.6227440 Richardson, I. E. G. (2010). The H.264 Advanced Video Compression
Standard (2nd ed., p. 316). John Wiley & Sons Ltd.