lec 10 video coding standard and system -...
TRANSCRIPT
Spring 2017: Multimedia Communication
Lec 10
Video Coding Standard and System - HEVC
Zhu Li
Course Web:
http://l.web.umkc.edu/lizhu/
Z. Li Multimedia Communciation, Spring 2017 p.1
Outline
Lecture 09 Video Signal Processing ReCap
Course Project Topic: Light Field Compression
Video Coding Standard and Systems
HEVC History
HEVC Performance
HEVC Key Coding Features
Z. Li Multimedia Communciation, Spring 2017 p.2
Coding Structure
Fast Block Motion Estimation:
Video Signal Processing
Z. Li Multimedia Communciation, Spring 2017 p.3
I P P P … P P P
……
GoP
1 2 3 4 5 6 7
I B B P B B P
GoP
MV coding modes in HEVC
3 MV coding modes
key ideas, predicting from current block’s spatio-temporal neighbors
new merge mode: saving MV coding bits by combine small blocks into larger blocks.
Z. Li Multimedia Communciation, Spring 2017 p.4
Intra Prediction in HEVC
Much more modes DC mode: copy DC values from neighbor
Planar mode: top row or left col average
Angular: pixels on certain line
Ref: Jani Lainema, Frank Bossen, Woojin Han, Junghye Min, Kemal Ugur, Intra Coding of the HEVC Standard. IEEE Trans. Circuits Syst. Video Tech. 22(12): 1792-1801 (2012)
Z. Li Multimedia Communciation, Spring 2017 p.5
Like a sparse transform basis!
Deblocking Filter
Reduce blocking artifact in the reconstructed frames
Can improve both subjective and objective quality
Filter in H.261:
[1/4, 1/2, 1/4]: Applied to non-block-boundary pixels in each block.
A low-pass smoothing filter.
In H.264 (and H.263v2), this is used in the prediction loop to
improve motion estimation accuracy. Decoder needs to do the
same. Also called loop filter.
Before…. and After
H.265: 8x4 block level
H.264: 4x4 block level
Z. Li Multimedia Communciation, Spring 2017 p.6
Sample Adaptive Offset (SAO) Filter
Classify pixels on block edge as one of the four categories
Offset its pixel value accordingly
Z. Li Multimedia Communciation, Spring 2017 p.7
Scalable Video Coding
Scalable video coding is desired in heterogeneous networks:
Types of scalability:
Z. Li Multimedia Communciation, Spring 2017 p.8
HW-3: Motion Estimation
Fractional Pel Motion Compensation
Use bi-linear interpolation to scale up the frames 2x2
Do ½ pel motion estimation and compensation
Compare block size and pel accuracy effects on the residual MSE
B-Frames
Implement a special version of B-frames, have forward and backward motion vectors
Compare the results with P frame motion estimation
Fast Motion Estimation
Implement Diamond search pattern, compare with the exhaustive search
Data set:
“foreman” sequence frames 150, 151, and 152.
Z. Li Multimedia Communciation, Spring 2017 p.9
Outline
Lecture 09 Video Signal Processing ReCap
Course Project Topic: Light Field Compression
Video Coding Standard and Systems
HEVC History
HEVC Performance
HEVC Key Coding Features
Z. Li Multimedia Communciation, Spring 2017 p.10
What is Light Field ?
Capturing all lights in an observing volume
Z. Li Multimedia Communciation, Spring 2017 p.11
Light Field Acquisition and Format
Lytro System
• Images were acquired using a Lytro Illum camera
• LFR (light field raw) file format as defined by Lytro contains the raw image
data:
Output of camera sensor, overlaid with ‘GRBG’ Bayer filter;
7728×5368 samples - 10-bit little endian integer values;
• A demosaicing process is required to obtain the color information from
the sensor data;
Demosaicing: raw data → RGB image;
From Prof. Fernando Pereira’s San Diego JPEG Presentation
Z. Li Multimedia Communciation, Spring 2017 p.12
View Extraction Processes
Compression Code Point:
• Light fields were extracted from the LFR files using the Light Field Toolbox
v0.4 made available by Donald Dansereau, QUT, AU;
• http://lightfield-forum.com/tag/lightfield-toolbox/
• The process of obtaining a specific view from the full light field involves
two main stages:
Coding
Multi-perspectives
Multi-focus
From Prof. Fernando Pereira’s San Diego JPEG Presentation
Z. Li Multimedia Communciation, Spring 2017 p.13
Lenslet (sensor) vs Sub-aperture (synthesis) Images
Z. Li Multimedia Communciation, Spring 2017 p.14
Lenslet sensor capture Sub-aperture image
Light Field Data Set
EPFL data set (6.6GB) : http://mmspg.epfl.ch/EPFL-light-field-image-dataset
Z. Li Multimedia Communciation, Spring 2017 p.15
a) Bikes b) Desktop c) Flowers
d) Friends 1 e) Magnets 1 f) Stone Pillars Outside
Light Field Compression Test
View Point Compression
Pre-selected 9 perspective view point in the light field
Z. Li Multimedia Communciation, Spring 2017 p.16
Naïve Solution – Direct Coding of Lenslet
R-D curves: Frenando Pereira/IST results:
Z. Li Multimedia Communciation, Spring 2017 p.17
JPEG JPEG 2000
H.264/AVC
Intra
HEVC
Intra
Naïve Solution – Codec Performances
JPEG, JPEG2K, AVC Intra, HEVC Intra
Fernando Pereira, at JPEG meeting in San Diego, 2016.02
Z. Li Multimedia Communciation, Spring 2017 p.18
Bikes Desktop Flowers
Re-Focusing Point Performance
Re-Focusing Operating Points
Z. Li Multimedia Communciation, Spring 2017 p.19
Re-Focusing Point R-D curves
Also across different codec, naïve solution (Fernando Pereira, IST)
Z. Li Multimedia Communciation, Spring 2017 p.20
JPEG JPEG 2000
H.264/AVC
Intra
HEVC
Intra
LF Coding Discussions
Redundancy in the lenslet images
Across lenslet prediction ?
o Optical correction ?
o Affine block matching ?
R-D optimization for certain view perspective ?
Across focal lens R-D optimization ?
Z. Li Multimedia Communciation, Spring 2017 p.21
Outline
Lecture 09 Video Signal Processing ReCap
Course Project Topic: Light Field Compression
Video Coding Standard and Systems
HEVC History
HEVC Performance
HEVC Key Coding Features
Z. Li Multimedia Communciation, Spring 2017 p.22
HEVC/H.265 History
HEVC Objective:
Z. Li Multimedia Communciation, Spring 2017 p.23
Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial
New Design Features
HEVC (H.265) vs AVC (H.264)
Z. Li Multimedia Communciation, Spring 2017 p.24
Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial
HEVC Development History
Long hard fight, many casualties…
Z. Li Multimedia Communciation, Spring 2017 p.25
HEVC Coding Pipeline
HEVC Diagram
Z. Li Multimedia Communciation, Spring 2017 p.28
Credit: Prof. Oscar Au, Tutorial on HEVC
HEVC Coding Structure
Quad Tree Decomposition:
Z. Li Multimedia Communciation, Spring 2017 p.29
Slide Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial
Ref:G. Schuster, PhD Thesis, 1996: Optimal Allocation of Bits Among Motion, Segmentation and Residual
HEVC Coding Structure
CTU- Coding Tree Unit
Upto 64x64 pel block
Macro Block equivalent
Aka LCU: Largest Coding Unit
CU – Coding Unit
Square NxN pel
Like Block
PU – Prediction Unit
Inter Prediction
Intra Prediction
Z. Li Multimedia Communciation, Spring 2017 p.30
Prediction Unit
PU:
Basic Unit for Transform & Quantization
Z. Li Multimedia Communciation, Spring 2017 p.31
Transform Unit (TU)
TU Square blocks for transform & quantization/coding, size: 4x4, 8x8,
16x16, 32x32
MaxTU size: 32 for luma, 16 for chroma
Min TU size: 4x4 for both luma and chroma
TU size can be larger than PU for inter PU case (MV merge)
Z. Li Multimedia Communciation, Spring 2017 p.32
Intra-Predicted Basis
As if it is a 1-non zero coefficient transform…
Z. Li Multimedia Communciation, Spring 2017 p.35
Ref:J. Laniema and W.-J. Han, “Intra Picture Prediction in HEVC”, Chapter in, Springer-Velag Book on High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014. Ed. V. Sze et. Al.
Pre-Processing for Intra Prediction
Smoothing filtering before intra prediction
Z. Li Multimedia Communciation, Spring 2017 p.36
Post-Processing for Intra Prediction
Remove discontinuity:
Z. Li Multimedia Communciation, Spring 2017 p.37
HEVC Intra as Still Image Coding Standard
Advancing the state or art by quite a bit
Z. Li Multimedia Communciation, Spring 2017 p.38
HEVC Inter Prediction
Supports 4x4 block ¼ pel accuracy Motion Compensation
Z. Li Multimedia Communciation, Spring 2017 p.39
Interpolation Filters
Quite a bit of memory and computing cost:
Z. Li Multimedia Communciation, Spring 2017 p.40
HEVC CABAC Throughput Improvement
Context Bypassing Mode:
Z. Li Multimedia Communciation, Spring 2017 p.43
Parallel Processing Tools: Slice/Tile
Z. Li Multimedia Communciation, Spring 2017 p.44
Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial
Profiles, Levels and Tiers
Operating HEVC in Profiles/Levels/Tiers
Z. Li Multimedia Communciation, Spring 2017 p.45
HEVC Resources
Main Spec: http://www.itu.int/ITU-T/recommendaBons/rec.aspx?rec=11885
T-CSVT Special Issue: 2012: Combined Issue on HEVC Standard and Research: http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6403920
2016: Special Issue on HEVC Extensions and Efficient HEVC Implementations:
http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=7372356
Springer Book: V. Sze, M. Budagavi, G. J. Sullivan (Editors), “High
Efficiency Video Coding (HEVC): Algorithms and Architectures,” Springer, 2014,http://www.springer.com/engineering/signals/book/978-3-319-06894-7
HM (open source software): https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSovware/
HARP: A python HM access tool: http://www.lms.lnt.de/HARP/
Z. Li Multimedia Communciation, Spring 2017 p.46
Summary
Light Field Compression
Sensor data at lenslet
Support new perspective and re-focusing
Interesting R-D optimization issue at new perspective and focusing point in the light field
Calls for new coding tools, based on HEVC intra, e.g,
HEVC
Crown Jewel of 20+ years of research and engineering
Significant performance gains over the previous state of art
Worthwhile to master this tool for a variety of research and biz opportunities.
Z. Li Multimedia Communciation, Spring 2017 p.47