roadmap

EE591f Digital Video Processing EE591f Digital Video Processing

11

RoadmapRoadmap

IntroductionIntroductionIntra-frame coding Intra-frame coding – Review of JPEGReview of JPEG

Inter-frame codingInter-frame coding– Conditional Replenishment (CR)Conditional Replenishment (CR)– Motion Compensated Prediction (MCP)Motion Compensated Prediction (MCP)

Object-based and scalable video coding*Object-based and scalable video coding*– Motion segmentation, scalability issuesMotion segmentation, scalability issues


22

Introduction to Video CodingIntroduction to Video Coding

Lossless vs. lossy data compressionLossless vs. lossy data compression– Source entropy H(X)Source entropy H(X)– Rate-Distortion function R(D) or D(R)Rate-Distortion function R(D) or D(R)

Probabilistic modeling is at the heart of data Probabilistic modeling is at the heart of data compressioncompression– What is P(X) for video source X?What is P(X) for video source X?– Modeling moving pictures is more difficult than Modeling moving pictures is more difficult than

modeling still images due to temporal dependencymodeling still images due to temporal dependency


33

Shannon’s PictureShannon’s Picture

Rate (bps)

Distortion

Coder ACoder B

Which coder wins, A or B?


44

Distortion MeasuresDistortion Measures

ObjectiveObjective– Mean Square Error (MSE)Mean Square Error (MSE)– Peak Signal-to-Noise-Ratio (PSNR)Peak Signal-to-Noise-Ratio (PSNR)– Measure the fidelity to original videoMeasure the fidelity to original videoSubjectiveSubjective– Human Vision System (HVS) basedHuman Vision System (HVS) based– Emphasize visual quality rather than fidelityEmphasize visual quality rather than fidelityWe only discuss objective measures in this We only discuss objective measures in this coursecourse


55

RoadmapRoadmap

IntroductionIntroductionIntra-frame codingIntra-frame coding – Review of JPEGReview of JPEG


Scalable video codingScalable video coding– 3D subband/wavelet coding and recent trend3D subband/wavelet coding and recent trend


66

A Tour of JPEG Coding Standard

Key Components

Transform

Quantization

Coding

-8×8 DCT-boundary padding

-uniform quantization-DC/AC coefficients

-Zigzag scan-run length/Huffman coding


77

JPEG Baseline Coder

169130173129

170181170183

179181182180

179180179179

169132171130

169183164182

179180176179

180179178178

167131167131

165179170179

177179182171

177177168179

169130165132

166187163194

17611615394

153183160183

Tour Example


88

Step 1: Transform• DC level shifting

• 2D DCT

169130173129

170181170183

179181182180

179180179179

169132171130

169183164182

179180176179

180179178178

167131167131

165179170179

177179182171

177177168179

169130165132

166187163194

17611615394

153183160183

412451

42534255

51535452

51525151

414432

41553654

51524851

52515050

393393

37514251

49515443

49494051

412374

38593566

48122534

25553655

-128

412451

42534255

51535452

51525151

414432

41553654

51524851

52515050

393393

37514251

49515443

49494051

412374

38593566

48122534

25553655

1342

1209

4021

1344

3055

4773

3046

3216

113916

109621

1793310

8101720

10242727

1326078

44131827

273856313

DCT


99

Step 2: Quantization

99103101120

100112121103

98958778

92726449

9211377103

1048110968

64555637

35242218

62805669

87515740

29222416

17141314

55606151

58264024

19141610

12121116

Q-table

1342

1209

4021

1344

3055

4773

3046

3216

113916

109621

1793310

8101720

10242727

1326078

44131827

273856313

0000

0000

0000

0000

0000

0000

0000

0000

0000

0001

1011

0111

0001

0123

2113

23520

Q

Why increasefrom top-left tobottom-right?


1010

Step 3: Entropy Coding

Zigzag Scan

0000

0000

0000

0000

0000

0000

0000

0000

0000

0001

1011

0111

0001

0123

2113

23520

(20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB)

Zigzag Scan

End Of the Block:All following coefficients are zero


1111

RoadmapRoadmap





1212

Conditional ReplenishmentConditional Replenishment

Based on motion detection rather than motion Based on motion detection rather than motion estimationestimationPartition the current frame into “still areas” Partition the current frame into “still areas” and “moving areas”and “moving areas”– Replenishment is applied to moving regions onlyReplenishment is applied to moving regions only– Repetition is applied to still regionsRepetition is applied to still regionsNeed to transmit the location of moving areas Need to transmit the location of moving areas as well as new (replenishment) informationas well as new (replenishment) information– No motion vectors transmittedNo motion vectors transmitted


1313

Conditional ReplenishmentConditional Replenishment


1414

Motion DetectionMotion Detection


1515

From Replenishment to PredictionFrom Replenishment to Prediction

Replenishment can be viewed as a degenerated Replenishment can be viewed as a degenerated case of predictioncase of prediction– Only zero motion vector is considered Only zero motion vector is considered – Discard the historyDiscard the history

A more powerful approach of exploiting A more powerful approach of exploiting temporal dependency is predictiontemporal dependency is prediction– Locate the best match from the previous frameLocate the best match from the previous frame– Use the history to predict the current Use the history to predict the current


1616

Differential Pulse Coded ModulationDifferential Pulse Coded Modulation

_

+D

xn

xnxn-1

yn

xn-1

+yn xn

D

EncoderDecoder

Q^yn

^

^^

^

xn-1^

^

Xn,yn: unquantized samples and prediction residues

Xn,yn: decoded samples and quantized prediction residues^ ^

nnnn yyxx ˆˆ


1717

Motion-Compensated Predictive CodingMotion-Compensated Predictive Coding


1818

A Closer LookA Closer Look


1919

Key ComponentsKey Components

Motion Estimation/CompensationMotion Estimation/Compensation – At the heart of MCP-based codingAt the heart of MCP-based coding

Coding of Motion Vectors (overhead)Coding of Motion Vectors (overhead)– Lossless: errors in MV are catastrophic Lossless: errors in MV are catastrophic

Coding of MCP residuesCoding of MCP residues– Lossy: distortion is controlled by the quantization Lossy: distortion is controlled by the quantization

step-sizestep-size

Rate-Distortion optimizationRate-Distortion optimization


2020

Block-based Motion ModelBlock-based Motion Model

Block sizeBlock size– Fixed vs. variableFixed vs. variable

Motion accuracyMotion accuracy– Integer-pel vs. fractional-pelInteger-pel vs. fractional-pel

Number of hypothesisNumber of hypothesis– Overlapped Block Motion Compensation (OBMC)Overlapped Block Motion Compensation (OBMC)– Multi-frame predictionMulti-frame prediction


2121

Quadtree Representation of Quadtree Representation of Motion Field with Variable BlocksizeMotion Field with Variable Blocksize


2222

Rate-Distortion Optimized BMARate-Distortion Optimized BMA

Distortion alone

Rate and Distortion

counted bits using a VLC table


2323

Experimental ResultsExperimental Results

Cited from G. Sullivan and L. Baker, “Rate-Distortion optimizedmotion compensation for video compression using fixed or variable size blocks”, Globecom’1991


2424

Fractional-pel BMAFractional-pel BMA

Recall the tradeoff between spending bits on Recall the tradeoff between spending bits on motion and spending bits on MCP residuesmotion and spending bits on MCP residuesIntuitively speaking, going from integer-pel to Intuitively speaking, going from integer-pel to fractional-pel is good for it dramatically fractional-pel is good for it dramatically reduces the variance of MCP residues for reduces the variance of MCP residues for some video sequence.some video sequence.The gain quickly saturates as motion accuracy The gain quickly saturates as motion accuracy refinesrefines


2525

8-by-8 block, half-pel, var(e)=123.88-by-8 block, integer-pel, var(e)=220.8

ExampleExample

MCP residue comparison for the first two frames of Mobile sequence


2626

Fractional-pel MCPFractional-pel MCP


2727

Multi-Hypothesis MCPMulti-Hypothesis MCP

Using one block from one reference frame Using one block from one reference frame represents a single-hypothesis MCPrepresents a single-hypothesis MCPIt is possible to formulate multiple hypothesis It is possible to formulate multiple hypothesis by consideringby considering– Overlapped blocksOverlapped blocks– More than one reference frameMore than one reference frameWhy multi-hypothesis?Why multi-hypothesis?– The benefit of reducing variance of MCP residues The benefit of reducing variance of MCP residues

outweighs the increased overhead on motionoutweighs the increased overhead on motion


2828

Example: B-frameExample: B-frame

fn-1 fn fn+1

1/5.0/0),,()1(),(),(ˆ 11 ayxfayxfayxf nnn


2929

Generalized B-frameGeneralized B-frame

fn-1 fn fn+1

nk

kk

n yxfayxf ),(),(ˆ

fn+2fn-2


3030

Multi-Hypothesis MCPMulti-Hypothesis MCP


3131


Motion Estimation Motion Estimation – At the heart of MCP-based codingAt the heart of MCP-based coding



step-sizestep-size



3232

Motion Vector CodingMotion Vector Coding

2D lossless DPCM2D lossless DPCM– Spatially (temporally) adjacent motion vectors are Spatially (temporally) adjacent motion vectors are

correlatedcorrelated– Use causal neighbors to predict the current oneUse causal neighbors to predict the current one– Code Motion Vector Difference (MVD) instead of Code Motion Vector Difference (MVD) instead of

MVsMVsEntropy coding techniquesEntropy coding techniques– Variable length codes (VLC)Variable length codes (VLC)– Arithmetic codingArithmetic coding


3333

MVD ExampleMVD Example

MV

MV1 MV2

MV3

),,( 321 MVMVMVmedianMVMVD

Due to smoothness of MV field, MVD usually hasa smaller variance than MV


3434

VLC Example VLC Example MVx/MVy symbol codeword

0

-1

-2

1

2

3

1

23456

1010011001000010100110

Exponential Golomb Codes: 0…01x…xm m


3535





step-sizestep-size



3636

MCP Residue CodingMCP Residue Coding

Transform Quantization Coding

Conceptually similar to JPEG

Transform: unitary transform

Quantization: Deadzone quantization

Coding: Run-length coding


3737

TransformTransform

Unitary matrix: A is real, AUnitary matrix: A is real, A-1-1=A=ATT

Unitary transform: A is unitary, Y=AXAUnitary transform: A is unitary, Y=AXATT

ExamplesExamples– 8-by-8 DCT8-by-8 DCT– 4-by-4 integer transform 4-by-4 integer transform

122111112112

1111

A


3838

Deadzone QuantizationDeadzone Quantization

2

0

deadzone

codewords


3939





step-sizestep-size



4040

Lagrangian Multiplier MethodLagrangian Multiplier MethodRDJ

motionmotiondfd RDJ

RECModeREC RDJ

Motion estimation

Mode selection

ModeMotion

2cQUANTMode

QUANT: a user-specified parameter controlling quantization stepsize


4141

SummarySummary

How does MCP coding work?How does MCP coding work?– The predictive model captures the slow-varying trend The predictive model captures the slow-varying trend

of the samples {fof the samples {fnn}}

– The modeling of prediction residues {eThe modeling of prediction residues {enn} is easier } is easier than that of original samples {fthan that of original samples {fnn}}

Fundamental weaknessFundamental weakness– Quantization error will propagate unless the memory Quantization error will propagate unless the memory

of predictor is refreshedof predictor is refreshed– Not suitable for scalable coding applicationsNot suitable for scalable coding applications


4242

RoadmapRoadmap





4343

Scalable vs. MulticastScalable vs. Multicast

What is scalable coding?What is scalable coding?

Multicast Scalable coding

foreman.yuv

foreman128k.codforeman256k.codforeman512k.codforeman1024k.cod

foreman.yuv

foreman.cod

1024512256128


4444

Spatial scalabilitySpatial scalability

11 00 11 11 11 …… 00 11 00 11 00 00 00 …… 11 11 00 11 00 00


4545

Temporal scalabilityTemporal scalability

11 00 11 11 11 …… 00 11 00 11 00 00 00 …… 11 11 00 11 00 00

Frame 0,1,2,3,4,5,…Frame 0,2,4,6,8,…Frame 0,4,8,12,…

30Hz15Hz7.5Hz


4646

SNR (Rate) scalabilitySNR (Rate) scalability

11 00 11 11 11 …… 00 11 00 11 00 00 00 …… 11 11 00 11 00 00

PSNRavg=30dB PSNRavg=35dB PSNRavg=40dB

N

iiavg PSNR

NPSNR

1

1PSNRi: PSNR of frame i


4747

Scalability via Bit-Plane CodingScalability via Bit-Plane Coding

A=(a0+a12+a222+ … … +a727)

Least Significant Bit (LSB)

Most Significant Bit (MSB)

Example A=129 sign=+,a0a1a2 …a7=10000001

sign=-, a0a1a2 …a7=00110011 A=-(4+8+64+128)=-204

sign bit


4848

Why DPCM Bad for Scalability?Why DPCM Bad for Scalability?

Base layer

Enhancement Layer 1

Enhancement Layer 2

Ibase P P P

Ienh1

Ienh2

1 2 3 …Frame number

P

P

P

P

P

P

suffer from drifting problemsuffer from coding efficiency loss


4949

3D Wavelet/Subband Coding3D Wavelet/Subband Coding

t

x

y

2D spatial WT+1D temporal WT


5050

Motion-Adaptive 3D Wavelet TransformMotion-Adaptive 3D Wavelet TransformRecall Haar transform

)12()2()(

)),12()2((21)(

nxnxnd

nxnxns

])[(21

],[

12

122

nnn

nnn

dWfs

fWfd

Motion-adaptive Haar transform

))()2((21)(

),12()2()(

ndnxns

nxnxnd

W,W-1: forward and backward motion vector

lifting-based implementation

roadmap

Documents

video codinglossless

video source x

quantization coding

current frame

motion segmentation

motion estimationpartition

prediction residuesxn

quantized prediction