video digitization and format conversionyao/ee4414/digital... · 2005. 9. 27. · digital video...

Video Digitization and Format ConversionVideo Digitization and Format ConversionVideo Digitization and Format ConversionVideo Digitization and Format Conversion

Yao WangPolytechnic University, Brooklyn, NY11201

© Yao Wang, 2003 Video digitization and format conversion

Outline

• How to capture digital video– Direct acquisition: digital video camcorder– Digitizing analog raster video– Digitizing analog film video (frame-based)

• Chrominance subsampling• Color coordinate • Digital video formats and applications• Video format conversion

– Deinterlacing– NTSC <-> PAL/SECAM– SDTV <-> HDTV

© Yao Wang, 2003 Video digitization and format conversion 3

A2D Conversion

• Digitization = Sampling + Quantization

xc(t) x[n] = xc(nT) �[ ]x n

Quanti-zation

Sampling

SamplingPeriod

T

QuantizationInterval

Q


Illustration of Sampling and Quantization

0 0.2 0.4 0.6 0.8 1

-1

-0.5

0

0.5

1T=0.1Q=0.25


Video Raster Revisited

Field 1 Field 2

Progressive Frame Interlaced Frame

How to convert a raster video to digital signal?

Horizontal retrace

Vertical retrace


Digitizing A Raster Video

• Digitization = Sampling + Quantization• Sample the raster waveform = Sample along the horizontal

direction• Apply the above sampling on Y,I,Q rasters separately• Quantize Y,I,Q samples to 8 bits each• How should we select the sampling rate?

– Must be faster than the Nyquist rate (twice the highest frequency)– For the samples to be aligned vertically, the sampling rate should

be multiples of the line rate– Horizontal sampling interval = vertical sampling interval (square

pixel)– Total sampling rate equal among different systems (525/30 vs

625/25MHz 5.13)(PAL/SECAM864(NTSC)858 === lls fff


BT.601* Video Format

480

lines

525

lines

122pel

16pel

858 pels720 pels

ActiveArea

525/60: 60 field/s57

6 lin

es

625

lines

864 pels

132pel

12pel

720 pels

ActiveArea

625/50: 50 field/s

* BT.601 is formerly known as CCIR601Pixels in non-shaded areas correspond to samples taken during horizontal/vertical retrace


Color Coordinate: YCbCr

• YCbCr = Digital equivalent of YUV– Cb = U = B-Y, Cr = V = R- Y– Each represented in 8 bits


RGB <--> YCbCr

Y_d = 0.257 R_d + 0.504 G_d + 0.098 B_d + 16,C_b = -0.148 R_d - 0.291 G_d + 0.439 B_d + 128,C_r = 0.439 R_d -0.368 G_d - 0.071 B_d + 128,

R_d = 1.164 Y_d’ + 0.0 C_b’+ 1.596 C_r’,G_d = 1.164 Y_d’ - 0.392 C_b’ -0.813 C_r’,B_d = 1.164 Y_d’ + 2.017 C_b’ + 0.0 C_r’,

Y_d’=Y_d -16, C_b’=C_b-128, C_r’=C_r-128


Sampling of Chrominance Components

• Human eye is less sensitive to chrominance• Chrominance signal has lower bandwidth

– Can be sampled at lower rate than Y


Chrominance Subsampling Formats

4:2:0For every 2x2 Y Pixels

1 Cb & 1 Cr Pixel(Subsampling by 2:1 bothhorizontally and vertically)

4:2:2 For every 2x2 Y Pixels

2 Cb & 2 Cr Pixel(Subsampling by 2:1

horizontally only)

4:4:4 For every 2x2 Y Pixels

4 Cb & 4 Cr Pixel(No subsampling)

Y Pixel Cb and Cr Pixel

4:1:1For every 4x1 Y Pixels

1 Cb & 1 Cr Pixel(Subsampling by 4:1

horizontally only)

I,Q raster are sampled at ½rate

I,Q raster are sampled at ¼rate

I,Q raster are sampled at the same rate as Y

Vertical down-sampling from 4:2:2


Digital Video Formats

Video Format Y Size Color Sampling

Frame Rate (Hz)

Raw Data Rate (Mbps)

HDTV Over air. cable, satellite, MPEG2 video, 20-45 Mbps SMPTE296M 1280x720 4:2:0 24P/30P/60P 265/332/664 SMPTE295M 1920x1080 4:2:0 24P/30P/60I 597/746/746 Video production, MPEG2, 15-50 Mbps BT.601 720x480/576 4:4:4 60I/50I 249 BT.601 720x480/576 4:2:2 60I/50I 166 High quality video distribution (DVD, SDTV), MPEG2, 4-10 Mbps BT.601 720x480/576 4:2:0 60I/50I 124 Intermediate quality video distribution (VCD, WWW), MPEG1, 1.5 Mbps SIF 352x240/288 4:2:0 30P/25P 30 Video conferencing over ISDN/Internet, H.261/H.263/MPEG4, 128-384 Kbps CIF 352x288 4:2:0 30P 37 Video telephony over wired/wireless modem, H.263/MPEG4, 20-64 Kbps QCIF 176x144 4:2:0 30P 9.1


Video Format Conversion

• Needed for various applications– Deinterlacing (interlaced -> progressive)– NTSC (525/50 ) <-> PAL (625/60)– SDTV <-> HDTV

• Using digital interpolation filters!– Filtering within one frame (spatial interpolation)– Filtering across frames for the same pixel (temporal

interpolation)– Spatial-temporal filtering


Why Deinterlacing

• Needed for – Display interlaced sequence in progressive monitor with

twice frame rate– Convert old TV material to SDTV/HDTV with good quality– Convert between NTSC and PAL with good quality– …


Deinterlacing Problem

From [Wang02]


Deinterlacing by Temporal Interpolation

• Field merging:– Merge two fields into one frame, repeats the frame twice– Equivalent to temporal interpolating using replication filter

• Field averaging– Averaging corresponding lines in preceding and following fields– Equivalent to temporal interpolation using linear averaging filter – Require two frame store !

• Both methods have problem when there are significant motion between two fields, especially moving vertical lines

• But can recover spatial details on stationary objects very well


100 100 50

100 100 100

100 100 100

100 100 100

100 100 50

100 50 50

100 100 50

100 50 50

100 100 50

100 50 50

100 100 50

100 100 50

100 50 50

100 50 50

100 50 50

100 100 50

100 100 100

100 100 50

100 100 100

100 100 100

Frame 1 Frame 2

Field 1 Field 2 Field 1 Field 2

An Example for Illustration


100 100 50

100 100 100

100 100 100

100 100 100

100 100 50

100 50 50

100 100 50

100 50 50

100 100 50

100 50 50

100 75 50

100 100 50

100 75 75

100 100 50

100 75 50

100 50 50

100 50 50

100 50 50

100 100 50

100 100 100

100 100 50

100 100 100

100 100 100

Frame 1 Frame 2


Deinterlacing by field averaging

Missing line3 = (field1 of frame1+field1 of frame2)/2


Deinterlacing by Spatial Interpolation

• Line averaging within same field– D=(C+E)/2– Equivalent to vertical interpolation using linear averaging

filter– Will not be able to recover fine details (e.g. alternating

horizontal lines)– Can recover moving vertical lines very well

• Higher order vertical filter can be used to improve quality

• Line/field averaging– D=(C+E+K+R)/4– The motion and spatial blur are both introduced but not as

obvious, on average may be better


100 100 50

100 100 100

100 100 100

100 100 100

100 100 50

100 50 50

100 100 50

100 50 50

100 100 50

100 50 50

100 100 50

100 100 50

100 100 50

100 100 50

100 100 50

100 50 50

100 50 50

100 50 50

100 100 50

100 100 100

100 100 50

100 100 100

100 100 100

Frame 1 Frame 2


Deinterlacing by line averaging

Missing line 3= (line2+line4)/2


100 100 50

100 100 100

100 100 100

100 100 100

100 100 50

100 50 50

100 100 50

100 50 50

100 100 50

100 50 50

100 88 50

100 100 50

100 88 62.5

100 100 50

100 88 50

100 50 50

100 50 50

100 50 50

100 100 50

100 100 100

100 100 50

100 100 100

100 100 100

Frame 1 Frame 2


Deinterlacing by field and line averaging

Missing line 3= (line2+line4+field1,frame1+field1,frame2)/4


Field 1 of Frame1

Field 2 of Frame1


Deinterlaced Field 2 by Merging Field 1 and Field 2


Deinterlaced Field Using Field Averaging

Deinterlaced Field Using Line Averaging


Deinterlaced Field Using Line and Field Averaging


Matlab Implementation

• Go through matlab script at class– deinterlacing.m


More Advanced Deinterlacing Methods

• Motion adaptive:– Switch between temporal/spatial interpolation depending on

motion detection results

• Edge adaptive– For spatial interpolation: interpolate along edge direction


PAL -> NTSC

13125=625*21=525*25 300=50*6=60*5

From [Wang02]


625 -> 525 lines

From [Wang02]


50 -> 60 fields

From [Wang02]


24 frames -> 60 fields

From [Wang02]


What you should know

• What are the factors considered when choosing the sampling rate for digitizing a raster video?

• Why should we sample chrominance components at lower sampling rate than the luminance component?

• What are the difference between 4:4:4, 4:2:2, 4:1:1, 4:2:0 colorformats?

• What are some of the common video formats and their intended applications?

• What is deinterlacing? What are some simple deinterlacingmethod (should be able to do calculation on example images and understand matlab scripts)

• What are some other format conversion problems? How are they accomplished?


References

• Y. Wang, J. Ostermann, Y. Q. Zhang, Video Processing and Communications, Prentice Hall, 2002. chapter 1, 3, 4. Mainly Sec. 1.5.

video digitization and format conversionyao/ee4414/digital... · 2005. 9. 27. · digital video...

Documents