enee631 digital image processing (spring'04) basics on video communications and other video...

ENEE631 Digital Image Processing (Spring'04)

Basics on Video Communications andBasics on Video Communications and

Other Video Coding Approaches/StandardsOther Video Coding Approaches/Standards

Spring ’04 Instructor: Min Wu

ECE Department, Univ. of Maryland, College Park

www.ajconline.umd.edu (select ENEE631 S’04) [email protected]

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

04

)

Based on ENEE631 Based on ENEE631 Spring’04Spring’04Section 13Section 13

ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [2]

Quick Review – A Few Basics on VideoQuick Review – A Few Basics on Video

Acquisition, Display, Analog & Digital FormatsAcquisition, Display, Analog & Digital Formats

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

04

)


Video CameraVideo Camera

Frame-by-frame capturing

CCD sensors (Charge-Coupled Devices)

– 2-D array of solid-state sensors– Each sensor corresponding to a pixel– Store in a buffer and sequentially read out– Widely used

small and light

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Video DisplayVideo Display

CRT (Cathode Ray Tube)– Large dynamic range– Bulky for large display

CRT physical depth has to be similar to screen width

LCD Flat-panel display– Use electrical field to change the optical properties hence the

brightness/color of liquid crystal– Generating the electrical field

by an array of transistors: active-matrix thin-film transistors by plasma

“Active-matrix display” (also known as TFT) has a transistor located at each pixel, allowing display be switched more frequently and less current to control pixel luminance. Passive matrix LCD has a grid of conductors with pixels located at the grid intersections

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Composite vs. Component VideoComposite vs. Component Video

Component video– Three separate signals for tristimulus color representation or luminance-

chrominance representation – Pro: higher quality– Con: need high bandwidth and synchronization

Composite video– Multiplex into a signal signal– Historical reason for transmitting color TV through monochrome

channel– Pro: save bandwidth– Con: cross talk

S-video: luminance sig. + single multiplexed chrominance sig.

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Analog Video RasterAnalog Video Raster

Line-by-line “Raster Scan”– Represent line-by-line image frame with 1-D analog

waveform– Synchronization signal for horizontal and vertical retrace

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Forming Picture on TV Tube (Monochrome)Forming Picture on TV Tube (Monochrome)

How many lines?

From B.Liu EE330S’01 Princeton

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


How Many TV Lines?How Many TV Lines?

Determined by spatial freq. response of HVS

dot

dot

Cannot resolve if

distance > 2000 x separation

(~ 0.03 degree viewing angle)

From B.Liu EE330S’01 Princeton

N = 500 for D=4H

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Review: Progressive vs. Interlaced scanReview: Progressive vs. Interlaced scanFrom B.Liu EE330S’01 Princeton

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Analog Color TV SystemsAnalog Color TV Systems

Historical notes – Color TV system had to be compatible with earlier monochrome TV system

3 formats– NTSC ~ North American + Japan/Taiwan – PAL ~ Western Europe + Asia(China) + Middle East– SECAM ~ Eastern Europe + France– What format in your home country?

From Wang’s Preprint Fig.1.5

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Comparison of Three Analog TV SystemsComparison of Three Analog TV Systems

– Spatial and temporal resolution– Color coordinate– Signal bandwidth– Multiplexing of luminance, chrominance, and audio

(From Wang’s Preprint)

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


NTSCNTSC

4:3 aspect ratio (width:height)

525 lines/frame, 2:1 interlace at field rate 59.94Hz– 483 active lines per frame; vertical retrace takes time of 9 lines– rest for broadcaster’s info. like closed caption

YIQ color coordinate for transmission– RGB primary slightly different from PAL– Orthogonal chrominance

I ~ orange-to-cyan; Q ~ green-to-purple (need less bandwidth)

Multiplexing over 6M Hz total bandwidth– Artifacts due to cross talk between luminance and chrominance

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


NTSC 6MHz Bandwidth NTSC 6MHz Bandwidth From Wang’s Preprint Fig.1.6(b)

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Analog Video RecordingAnalog Video Recording

Comparison of common formats

From Wang’s Preprint Table 1.2

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Digital Video FormatsDigital Video Formats ITU-R BT.601 recommendation

Downsampled chrominance– Y Cb Cr coordinate and four subsampling formats

Inter. Telecomm. Union – Radio sector

Wang’sPreprint Fig.1.8

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


From Wang’sPreprint

Table 1.3

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


ResourceResource

Background and Motivation on Background and Motivation on Multimedia Coding / CommunicationsMultimedia Coding / Communications

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

04

)


Generations of Video CodingGenerations of Video Coding

UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)

From R.Liu Seminar Course ’00 @ UMCP


Channel Bandwidth Channel Bandwidth

UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)



UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)

Storage CapacityStorage Capacity

UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)



Source Video FormatsSource Video Formats

UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)



Application RequirementsApplication Requirements

UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)



Other Standard and Considerations for Other Standard and Considerations for

Digital Video Coding Digital Video Coding

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

04

)


Performance Tradeoff for Video CodingPerformance Tradeoff for Video Coding

From R.Liu’s Handbook Fig.1.2:

“mos” ~ 5-pt mean opinion scale of bad, poor, fair, good, excellent

UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)


H.26x for Video TelephonyH.26x for Video Telephony Remote face-to-face communication: A dream for years

H.26x – Video coding targeted low bit rate– Through ISDN or regular analog telephone line ~ on the order of 64kbps – Need roughly symmetric complexity on encoder and decoder

H.261 (early 1990s)– Similar to simplified MPEG-1 ~ block-based DCT/MC hybrid coder– Integer-pel motion compensation with I/P frame only ~ no B frames– Restricted picture size/fps format and M.V. range

H.263 (mid 1990s) and H.263+/H.263++ (late 1990s)– Support half-pel motion compensation & many options for improvement

H.264 (latest, 2001-): also known as H.26L / JVT / MPEG4 part10

– Hybrid coding framework with many advanced techniques– Focusing on greatly improving compression ratio at a cost of complexity

UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)


MPEG-2MPEG-2

Extend from MPEG-1

Target at high-resolution high-bit-rate applications

– Digital video broadcasting, HDTV, …– Also used for DVD

Support scalability

Support interlaced video

– Frame pictures vs. Field pictures– New prediction modes for motion compensation related to interlaced

video Use previously encoded fields to do M.E.-M.C.U

MC

P E

NE

E4

08

G S

lide

s (c

rea

ted

by

M.W

u &

R.L

iu ©

20

02

)


Scalability in Video CodecsScalability in Video Codecs

Scalability: provide different quality in a single stream– Stack up more bits on base layer to provide improved quality

Possible ways for achieving scalabilities– SNR Scalability ~ Multiple–quality video services

Basic vs. premium quality

– Spatial Scalability ~ Multiple-dimension displays Display on PDA vs. PC vs. Super-resolution display

– Temporal Scalability ~ Multiple frame rates– Frequency Scalability ~ Blurred version to sharp, detailed version

Layered coding concept facilitates:– Unequal error protection – Efficient use of resources– Different needs from customers – Multiple services

UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)


SNR ScalabilitySNR Scalability

Two layers with same spatio-temporal resolution but different qualities

base-layerencoder

base-layerdecoder

enhancement-layerencoder

mul

tipl

exer+ -

Video inBase-layerbitsteam

Enhancement-layerbitsteam

Outputbitsteam

From R.Liu Seminar Course @ UMCP


MPEG-4MPEG-4

Many functionalities targeting a variety of applications

Introduced object-based coding strategy– For better support of interactive applications & graphics/animation video– Require encoder to perform object segmentation

difficult for general applications

Introduced error resilient coding techniques– “Streaming video profile” for wireless multimedia applications

Part-10 is converged into H.264– Focused on improving compression ratio and error resilience– Stick with Hybrid coding frameworkU

MC

P E

NE

E4

08

G S

lide

s (c

rea

ted

by

M.W

u &

R.L

iu ©

20

02

)


Object-based Coding in MPEG-4Object-based Coding in MPEG-4

Interactive functionalities Higher compression

efficiency by separately handling – Moving objects– Unchanged background– New regions– M.C.-failure regions=> “Sprite” encoding

Object segmentationneeded (not easy )– Based on color, motion,

edge, texture, etc.– Possible for targeted

applications

Revised from R.Liu Seminar Course @ UMCPU

MC

P E

NE

E4

08

G S

lide

s (c

rea

ted

by

M.W

u &

R.L

iu ©

20

02

)


Analysis-Synthesis Coding: A General FrameworkAnalysis-Synthesis Coding: A General Framework

From R.Liu Seminar Course @ UMCP


MPEG-7MPEG-7

“Multimedia Content Description Interface”

– Not a video coding/compression standard like previous MPEG– Emphasize on how to describe the video content for efficient

indexing, search, and retrieval

Standardize the description mechanism of content

– Descriptor, Description Scheme, Description Definition Languages– Example of MPEG-7 visual descriptor: Color, Texture, Shape, …

Figure from MPEG-7 Document N4031 (March 2001)

UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)

enee631 digital image processing (spring'04) basics on video communications and other video...

Documents

lightumcp enee631 slides

line image frame

tv lines

transmitting color tv

analog tv systemsspatial

tv tube monochromehow

pixel luminance

active lines