video - sharif university of technologyce.sharif.edu/courses/89-90/2/ce873-1/resources/root... ·...

Video (Fundamentals, Compression Techniques & Standards)

Hamid R. RabieeMostafa Salehi, Fatemeh Dabiran, Hoda Ayatollahi

Spring 2011

Digital Media Lab - Sharif University of Technology2

Outlines

² Frame Types

² Color

² Video Compression Techniques

² Video Coding Standards² H.261

² H.263

² MPEG Family

Digital Video

² Video; a multi-dimensional signal

² Video is a sequence of 2D images called frames. Digital video is digitized

version of a 3D function f(x,y,t) .

3

Frame N-1

Frame 0

time

Digital Media Lab - Sharif University of Technology

Frame Types

² I-frames are coded without reference to other frames. Serve as reference

pictures for predictive-coded frames.

² P-frames are coded using motion compensated prediction from a past I-

frame or P-frame.

² B-frames are bidirectionally predictive-coded. Highest degree of

compression, but require both past and future reference pictures for motion

compensation.

² D-frames are DC-coded. Of the DCT coefficients only the DC coefficients

are present. Used in interactive applications like VoD for rewind and fast-

forward operations.

4

Frames display order : I B Frames display order : I B BB P B P B BB P B P B BB I I ……A GroupA Group--OfOf--PicturePicture


I-frames Compression


Motion Compensation

² Exploits temporal redundancy in video frames

² Prediction:² Assumes that “locally” current frame can be modeled as a translation of a previous

frame

² Displacement need not be the same everywhere in the frame à encode motion

information properly for accurate reconstruction

² Bi-directional Interpolation:² High degree of compression

² Areas just uncovered are not predictable from the past, but can be predicted from the future

² Effect of noise and errors can be reduced by averaging between past and future references

² Frequency of B-frames: increasing the frequency of B-frames

² Improves compression efficiency, but …

² Decreases the correlation between the B-frame and the references, as well as between the references

à reasonable to space references by 1/10th of a second


Motion Compensation


P-frame Compression


Why do we need B-frames?

² Bi-directional prediction works better than only using previous frames

when occlusion occurs.


For this example, the prediction from nextframe is used and the prediction from previousframe is not considered.

B-frames Compression


B-frame Advantage

² B-frames increase compression.

² Typically use twice as many B frames as I+P frames.


B-frame Disadvantages

² Computational complexity.

² More motion search, need to decide whether or not to average.

² Increase in memory bandwidth.

² Extra picture buffer needed.

² Need to store frames and encode or playback out of order.

² Delay

² Adds several frames delay at encoder waiting for need later frame.

² Adds several frames delay at decoder holding decoded I/P frame, while

decoding and playing prior B-frames that depend on it.


Color

² Human Eye has receptors for brightness (in low light), and separate

receptors for red, green, and blue.

² Can make any color we can see by mixing red, green and blue light in

different intensities


Color TV

² Original TV standards were black and white.

² AM: Amplitude of signal determines brightness.

² How to add color without changing TV transmitters, and in such a way that

it’s compatible with existing B&W TVs?

² Add a high frequency subcarrier in band within B&W TV signal.

² Not noticeable on B&W TV - would show as high frequency pattern, but human eye

can’t really see this well.

² Modulate the phase of the sub carrier to indicate the color.

² Problem: how to calibrate the absolute phase.

² Get this wrong, and the colors display incorrectly.


NTSCNational Television Standard Committee

² Introduced in 1953 (in US)

² Used in US, Canada, Japan

² 30 frames per second (Actually 29.97)² Interlaced (even/odd field lines), so 60 fields per second.

² Same as 60Hz AC power in these countries

² 525 lines² Picture only on 480 of these => 640x480 monitors

² Rest are the vertical rescan.

² Aspect ratio is 4:3

² Needs color calibration

² Uses a color burst signal at start of each line, but needs TV to be adjusted relative to

this. “NTSC = Never Twice Same Color”


PALPhase Alternating Line

² Introduced in 1967 (by Walter Bruch in Germany)

² PAL-I(UK), PAL-B/G (much of Europe), PAL-M (Brazil) …² Differ mainly in audio subcarrier frequency.

² 25 frames per second² Interlaced (even/odd field lines), so 50 fields per second.

² Same as 50Hz AC power in these countries

² 625 lines² Only 576 lines used for picture.

² Rest are vertical retrace, but often carry teletext information.

² Color phase is reversed on every alternate line.² Originally human eye would average to derive correct color.

² Now TV sets auto calibrate to derive correct color.


SÉCAMSéquentiel Couleur Avec Mémoire

² Introduced in 1967 (in France)

² “System Essentially Contrary to American Method”

² Used in France, Russia, Eastern Europe…

² 625 lines, 25 fps interlaced, like PAL

² Uses FM modulation of subcarrier.

² Red-Luminance difference on one line

² Blue-Luminance difference on next line

² Uses a video line store to recombine the two signals

² Vertical color resolution is halved relative to NTSC and PAL.

² Human eye is not sensitive to lack of spatial color information.


Colorspace Representations

² RGB (Red, Green, Blue)

² Basic analog components (from camera/to TV tube)

² YPbPr (Y, B-Y, R-Y)

² Color space derived from RGB used in component video. Y= Luminance, B =

Blue, R = Red

² YUV

² Similar to YPbPr but scaled to be carried on a composite carrier.

² YCbCr Digital representation of YPbPr colorspace (8 bit, twos complement)


Color System in Video

² YUV was used in PAL (an analog video standard) and also for digital

video.

² Y is the luminance component (brightness)

Y = 0.299 R + 0.587 G + 0.144 B

² U and V are color components

U = B – Y

V = R - Y

Y U V21 Digital Media Lab - Sharif University of Technology

RGB vs. YUV


YUV Formats

² YUV 4:4:4

² 8 bits per Y,U,V channel (no chroma down sampling)

² YUV 4:2:2

² 4 Y pixels sample for every 2 U and 2V

² 2:1 horizontal down sampling, no vertical down sampling

² YUV 4:2:0

² 2:1 horizontal down sampling

² 2:1 vertical down sampling

² YUV 4:1:1

² 4 Y pixels sample for every 1 U and 1V

² 4:1 horizontal down sampling, no vertical down sampling


Color System in Video

² YIQ is the color standard in NTSC.

I Q

24 Digital Media Lab - Sharif University of Technology

Digital Video Formats

² Common Intermediate Format (CIF):

² This format was defined by CCITT (TSS) for H.261 coding standard

(teleconferencing and videophone).

² Several size formats:

² SQCIF: 88x72 pixels.

² QCIF: 176x144 pixels.

² CIF: 352x288 pixels.

² 4CIF: 704x576 pixels.

² Non-interlaced (progressive), and chrominance sub-sampling using 4:2:0.

² Frame rates up to 25 frames/sec


Digital Video Formats

² Source Input Format (SIF):

² Utilized in MPEG as a compromise with Rec. 601.

² Two size formats (similar to CIF):

² QSIF: 180x120 or 176x144 pixels at 30 or 25 fps

² SIF: 360x240 or 352x288 pixels at 30 or 25 fps

² Non-interlaced (progressive), and chrominance sub-sampling using 4:2:0.

² High Definition Television (HDTV):

² 1080x720 pixels.

² 1920x1080 pixels.


Uncompressed Video Data Rate

² Examples (CCIR 601)

² PAL signal: 864x625, YUV 4:2:2 20 bits/pixel, 25fps. 270Mb/s

² PAL signal: 864x625, YUV 4:2:2 16 bits/pixel, 25fps. 216Mb/s

² PAL video: 720x576, YUV 4:2:2 16 bits/pixel, 25fps. 166Mb/s (~1GByte/min)

² Firewire: 400Mb/s (800Mb/s)

² USB 2.0: 480Mb/s


VIDEO COMPRESSION REVIEW


Need for Compression

² Large data rate and storage capacity requirement

² Compression algorithms exploit:

² Spatial redundancy (i.e., correlation between neighboring pixels)

² Spectral redundancy (i.e., correlation between different frequency spectrum)

² Temporal redundancy (i.e., correlation between successive frames)


600 MB/image180x180 km230 m2 resolution

Satelliteimagery

30 Mbytes/s30 frames/s,640x480 pixels,3 bytes/pixel

NTSC video

Requirement for Compression Algorithms

² Objectives² Minimize the complexity of the encoding and decoding process

² Ensure a good quality of decoded images

² Achieve high compression ratios

² Other general requirements² Independence of specific size and frame rate

² Support various data rates


Classification of Compression Algorithms

² Lossless compression² Reconstructed image is mathematically equivalent to the original image

(i.e., reconstruction is perfect)

² Drawback: achieves only a modest level of compression (about a factor of

5)

² Lossy compression² Reconstructed image demonstrates degradation in the quality of the image

àthe techniques are irreversible

² Advantage: achieves very high degree of compression (compression ratios

up to 200)

² Objective: maximize the degree of compression while maintaining the

quality of the image to be virtually lossless


Compression Techniques: Fundamentals

² Entropy encoding

² Ignores semantics of input data and compresses media streams by regarding

them as sequences of bits

² Examples: run-length encoding, Huffman encoding, ...

² Source encoding

² Optimizes the compression ratio by considering media specific characteristics

² Examples:

² Predictive coding: e.g., DPCM

² Layered coding: e.g., bit-plane coding, sub-sampling

² Transform coding: e.g., DCT, FFT, Wavelet, ...

² Most compression algorithms employ a hybrid of the above techniques


Entropy Coding

² Run-length encoding

α α α α →4α

² Huffman encoding

² Employ variable length codes

² Assign fewer bits to encode more frequently occurring values à exploit the

statistical distribution of the values within an data sequence

² Share codebook between encoder and decoder


Source Coding: Predictive Coding

² Basic technique

² Predict the value at a pixel by using the values of the neighboring pixels; and

² Encode the difference between the actual value and the predicted value

² Predictor

² Dimension of the predictor

² Order of the predictor: number of pixels used

² Example of a third order predictor:

² Huffman encoding of differential images


Source Coding: Bit-plane Encoding

² An N *N image with k bits per pixel can be viewed as k N *N bit planes

² Encode each bit plane separately

² Advantages:

² Permits progressive transmission of encoded images (most significant bit plane

first - since it generally contains more information)

² Encoding should be carried out such that separate encoding yields better

performance than jointly encoding the bit Planes

² Gray codes are better suited as compared to binary encoding


Source Coding: Transform Coding

² Subdivide an individual N x N image into several n x n blocks

² Each n x n block undergoes a reversible transformation

² Basic approach:

² De-correlate the original block à radiant energy is redistributed amongst only a

small number of transform coefficients

² Discard many of the low energy coefficients (through quantization)

² General requirements:

² Image independence

² Should be computationally efficient


VIDEO CODING STANDARDS


Video Coding Standards

² H.261

² H.263

² H.263+

² MPEG Family

² MPEG-1

² MPEG-2

² MPEG-4

² MPEG-7

² H.264


H.261

² H.261 is an ITU video compression standard finalized in 1990.

² The basic scheme of H.261 has been retained in the newer video standards.

² H.261 supports bit rates at p*64 kbps (p=1..30).


H.261

² In H.261, motion vectors are in the range [-15,15]x[-15,15]

² H.261 uses a constant step-size for different DCT coefficients.

² For DC coefficients

² For AC coefficients

Where scale = 1 .. 31


Group of macroBlocks (GOB)

² To reduce the error propagation problem, H.261 makes sure that a

“group” of Macro-Blocks can be decoded independently.


H.261 Bit Stream Syntax


H.263

² H.263 is an improved video coding standard for video conferencing

through PSTN (public switching telecommunication network).

² Apart from QCIF and CIF, it supports SubQCIF, 4CIF and 16CIF.

² H.263 has a different GOB scheme.


H.263 Motion Compensation

² The motion compensation in this standard is a bit different from the

MPEG method!

² The motion compensation in the core H.263 is based on one motion vector per

macroblock of 16 × 16 pixels, with half pixel precision.


Motion vector prediction

Motion vector prediction for the border macroblocks

H.263+

² H.263 Ver. 2 (H.263+), ITU-T² Additional negotiable options for H.263.

² New features include: deblocking filter, scalability, slicing for network

packetization and local decode, square pixel support, arbitrary frame size,

chromakey transparency, etc…

² Arbitrary frame size, pixel aspect ratio (including square), and picture clock

frequency

² Advanced INTRA frame coding

² Loop de-blocking filter

² Slice structures

² Supplemental enhancement information

² Improved PB-frames


MPEG

² Moving Picture Experts Group

² JPEG does not exploit temporal (i.e., frame-to-frame) redundancy present in all video

sequences

² MPEG exploits temporal redundancy

² MPEG requirements:² Random access

² Fast searches - both forward and reverse

² Reverse playback

² Audio-video synchronization

² Robustness to errors

² Low encoding/decoding delay

² Editability


MPEG Family

² MPEG-1

² Similar to H.263 CIF in quality

² MPEG-2

² Higher quality: DVD, Digital TV, HDTV

² MPEG-4/H.264

² More modern codec.

² Aimed at lower bitrates.

² Works well for HDTV too.


MPEG-1 Video

² MPEG-1 was approved by ISO and IEC in 1991 for “Coding of Moving

Pictures and Associated Audio for Digital Storage Media at up to about

1.5Mbps”.

² MPEG-1 standard is composed of

² System

² Video

² Audio

² Conformance

² And Software


MPEG-1 Standard - An Overview

² Two categories: intra-frame and inter-frame encoding

² Contrasting requirements: delicate balance between intra- and inter-

frame encoding

² Need for high compression à only intra-frame encoding is not sufficient

² Need for random access à best satisfied by intra-frame encoding

² Overview of the MPEG algorithm:² DCT-based compression for the reduction of spatial redundancy (similar to

JPEG)

² Block-based motion compensation for exploiting the temporal redundancy

² Motion compensation using both causal (predictive coding) and non causal

(interpolative coding) predictors


Exploiting Temporal RedundancyThree types of frames in MPEG-1

² I-frames:

² Intra-coded frames, provide access

points for random access – yield

moderate compression

² P-frames:

² Predicted frames are encoded with

reference to a previous I or P frame

² B-frames:

² Bi-directional frames encoded using

the previous and the next I/P frame

² Achieves maximum compression


Example

² The figure illustrates the relationship between these three types of picture.

Since B-pictures use I and P-pictures as predictions, they have to be coded

later. This requires reordering the incoming picture order, which is

carried out at the preprocessor.


Motion Representation

² 16 x 16 blocks used as motion-compensation units (referred to as macro blocks)² Macro block size selected based on the tradeoff between the gain due to motion compensation and

the cost of coding motion information

² Types of macro blocks:² Intra, forward-predicted, backward-predicted, average

² Two types of information are maintained:² Motion vector:

² The difference between the spatial locations of the macro blocks

² One motion vector for forward/backward predicted blocks, and two vectors for average blocks

² Adjacent motion vectors typically differ only slightly encode them using differential encoding

techniques (e.g., DPCM)

² Difference between the macro block being encoded and its predictor block(s) - encode the difference

using DCT-based transform coding techniques


Motion Estimation

² Block-matching techniques employed for motion estimation

² Motion vector obtained by minimizing the mismatch between the block

being encoded and its predictor

² Exhaustive search for such a block yield good results – but the complexity

can be prohibitive

² Tradeoff between the quality of the motion vector versus the complexity of

the motion estimation process is left to the implementer


Difference of MPEG-1 with H.261

² Picture formats (SIF vs. CIF)

² GOB structure


Slices in MPEG-1

Difference of MPEG-1 with H.261 (cont)

Intra-coding quantization table Inter-coding quantization table

Intra mode:

Inter mode:

Scale=1..31

(the prediction error is like noise and their DCT coefficients are quite “flat”. We can use a uniform quantization table.)

MPEG-1 uses different quantization tables for I and P or B frames.


Difference of MPEG-1 with H.261 (cont)

² Sub pixel motion estimation in MPEG-1.

² Motion range up to 512 pixels.

² MPEG adds another layer called “Group Of Pictures” (GOP) to allow

random video access.


MPEG-1 Video Stream


MPEG-2

² MPEG-2 profiles and levels:


Profiles and Levels in MPEG-2

Scalable Layered Coding

² Need for Hierarchical Coding/Scalable Compression

² Facilitate access to images at different quality levels or resolutions

² Progressive transmission:

² Transmit image information in stages; at each stage, the reconstructed image is

progressively improved

² Motivated by the need for transmitting images over low bandwidth channels

² Permits progressive transmission to be stopped either if an intermediate version is of

satisfactory quality or the image is found to be of no interest

² Examples: multimedia databases, tele-browsing, etc.

² Multi-use environments:

² Support a number of display devices with differing resolutions

² Optimizes utilization of storage server and network resources

² Example: video-in-a-window archical Coding/Scalable Compression


Scalability

² SNR scalability² Base layer uses rough quantization, while enhancement layers encode the

residue errors.

² Spatial scalability² Base layer encodes a small resolution video; enhancement layers encode the

difference of bigger resolution video with the “un-sampled” lower resolution

one.

² Temporal scalability² Base layer down-samples the video in time; enhancement layers include the rest

of the frames.

² Hybrid scalability


Scalability Example


Spatial Scalability

SNR Scalability

MPEG-2 vs. MPEG-1

² Sequence layer:

² progressive vs. interlaced

² More aspect ratios (e.g. 16x9)

² Syntax can now signal frames sizes up to 16383x16383

² Pictures must be a multiple of 16 pixels

² MPEG-2 can use a modified zig-zag for run-length encoding of the

coefficients:


MPEG-2 vs. MPEG-1

² Picture Layer:

² All MPEG-2 motion vectors are always half-pixel accuracy

² MPEG-1 can opt out, and do one-pixel accuracy.

² DC coefficient can be coded as 8, 9, 10, or 11 bits.

² MPEG-1 always uses 8 bits.

² Optional non-linear macroblock quantization, giving a more dynamic step size

range:

² 0.5 to 56 vs. 1 to 32 in MPEG-1.

² Good for high-rate high-quality video.


Interlacing

² Although MPEG-2 only codes full frames (both fields), it support both

field prediction and frame prediction for interlaced sources.

² The current uncompressed frame has two fields.

² Can do the motion search independently for each field.

² Half the lines use one motion vector and half use the other to produce the

reference block.


MPEG-4

² ISO/IEC designation 'ISO/IEC 14496’: 1999

² MPEG-4 Version 2: 2000

² Aimed at low bitrate (10Kb/s)

² Can scale very high (1Gb/s)

² Based around the concept of the composition of basic video objects into a

scene.


MPEG-4

² Initial goal of MPEG-4

² Very low bit rate coding of audio visual data.

² MPEG-4 (at the end)

² Officially up to 10 Mbits/sec.

² Improved encoding efficiency.

² Content-based interactivity.

² Content-based and temporal random access.

² Integration of both natural and synthetic objects.

² Temporal, spatial, quality and object-based scalability.

² Improved error resilience.


Audio-Video Object


MPEG-4 Standard

² Defines the scheme of encoding audio and video objects

² Encoding of shaped video objects.

² Sprite encoding.

² Encoding of synthesized 2D and 3D objects.

² Defines the scheme of decoding media objects.

² Defines the composition and synchronization scheme.

² Defines how media objects interact with users.


MPEG-7 Standard

² (2001) MPEG7, ISO “Content Representation for Info Search”

² Specify a standardized description of various types of multimedia

information. This description shall be associated with the content itself, to

allow fast and efficient searching for material that is of a user’s interest.

² Mpeg-7

² Independent of the coding format of the media & physical location of the

media

² Facilitates searching for media content with ease

² Mpeg-7 supports

² text-based queries

² complex content-based queries.


Structure of the standard

² Mpeg-7 standardizes a representation of meta-data

² Media content has metadata

² Metadata provides context for data

² Normative representations and semantics of metadata is common in

MPEG-7


MPEG-7 Applications

² Storage and retrieval of audiovisual databases (image, film, radio archives)

² Broadcast media selection (radio, TV programs)

² Surveillance (traffic control, surface transportation, production chains)

² E-commerce and Tele-shopping (searching for clothes / patterns)

² Remote sensing (cartography, ecology, natural resources management)

² Entertainment (searching for a game, for a karaoke)

² Cultural services (museums, art galleries)

² Journalism (searching for events, persons)

² Personalized news service on Internet (push media filtering)

² Intelligent multimedia presentations

² Educational applications

² Bio-medical applicationsDigital Media Lab - Sharif University of Technology71

Why do we need MPEG-7 ?


H.264 (MPEG-4, Part 10)

² MPEG-4, Part 10 is also known as H.264.

² Advanced video coding standard, finalized in 2003.


H.264 vs. MPEG-2

² Multi-picture motion compensation.² Can use up to 32 different frames to predict a single frame.

² B-frames in MPEG-2 only code from two.

² Variable block-size motion compensation² From 4x4 to 16x16 pixels.

² Allows precise segmentation of edges of moving regions.

² Quarter-pixel precision for motion compensation.

² Weighted prediction (can scale or offset predicted block)² Useful in fade-to-black or cross-fade between scenes.

² Spatial prediction from the edges of neighboring blocks for "intra“ coding.

² Choice of several more advanced context-aware variable length coding schemes (instead

of Huffman).


H.264 Performance

² Typically half the data rate of MPEG-2.

² HDTV:

² MPEG-2: 1920x1080 typically 12-20 Mbps

² H.264: 1920x1080 content at 7-8 Mbps


H.264 Usage

² Pretty new, but expanding use.

² Included in MacOS 10 (Tiger) for iChat video conferencing.

² Used by Video iPod.

² Adopted by 3GPP for Mobile Video.

² Mandatory in both the HD-DVD and Blu-ray specifications for High Definition

DVD.


Video Standards Applications

² H.261, ITU-T

² Designed to work at multiples of 64 kb/s (px64).

² MPEG-1, ISO “Storage & Retrieval of Audio & Video”

² Main application is CD-ROM based video (~1.5 Mb/s).

² MPEG-2, ISO “Digital Television”

² Main application is video broadcast (DirecTV, DVD, HDTV).

² Typically operates at data rates of 2-3 Mb/s and above.

² H.263, ITU-T

² Evolution of all of the above.

² Targeted low bit rate video <64 kb/s. Works well at high rates, too.


Video Standards Applications

² H.263 Ver. 2 (H.263+), ITU-T

² Additional negotiable options for H.263.

² MPEG-4, ISO “Multimedia Applications”

² Support for multi-layered, non-rectangular video display

² MPEG7, ISO “Content Representation for Info Search”



Next Session

NGN

79

80

Any Question

Thank you!Winter 2011


video - sharif university of technologyce.sharif.edu/courses/89-90/2/ce873-1/resources/root... ·...

Documents