sampling issues in image and video z – 2-d sampling at ... · pdf filesampling issues in...

11

ENEE631 Digital Image Processing (Spring'06)

Sampling Issues in Image and VideoSampling Issues in Image and Video

Spring ’06 Instructor: K. J. Ray Liu

ECE Department, Univ. of Maryland, College Park

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

04)

ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [2]

Overview and LogisticsOverview and Logistics

Last Time:– Motion analysis– Geometric relations and manipulations

Today:– 2-D sampling at Rectangular grid– Lattice theory for multidimensional sampling at non-rectangular grid– Sampling and resampling for video

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

04)


Sampling: From 1Sampling: From 1--D to 2D to 2--D and 3D and 3--DD

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

04)


Review: 1Review: 1--D SamplingD SamplingTime domain– Multiply continuous-time signal with periodic impulse train

Frequency domain– Duality: sampling in one domain tiling in another domain

FT of an impulse train is an impulse train (proper scaling & stretching)

Review Oppenheim “Sig. & Sys”Chapt.7 (Sampling)Chapt.3,4,5 (FS,FT,DFT)

x(t)

p(t) = Σk δ ( t - kT)T

xs(t)

P(ω) = Σ k δ ( ω - 2kπ/T) *2π/T

2π/TX(ω)

ω

Xs(ω)

2π/T

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01)

22


Review: 1Review: 1--D Sampling TheoremD Sampling Theorem

1-D Sampling Theorem– A 1-D signal x(t) bandlimited within [-ωB,ωB] can be uniquely

determined by its samples x(nT) if ωs > 2ωB (sample fast enough).– Using the samples x(nT), we can reconstruct x(t) by filtering the

impulse version of x(nT) by an ideal low pass filter

Sampling below Nyquist rate (2ωB) cause Aliasing

Xs(ω) with ωs < 2ωB Aliasing

ωs=2π/T

ωB

Xs(ω) with ωs > 2ωB Perfect Reconstructable

ωs=2π/T

ωB-ωs

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01)


Extend to 2Extend to 2--D Sampling with Rectangular GridD Sampling with Rectangular Grid

Bandlimited 2-D signal– Its FT is zero outside a bounded region ( |ζx|> ζx0, |ζy|> ζy0 ) in

spatial freq. domain– Real-word multi-dimensional signals often exhibit diamond or

football shape of supportWith spectrum normalization, we will get spherical shape of support

Jain’s Fig.4.6

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01/2

004)


22--D Sampling (contD Sampling (cont’’d)d)2-D Comb functioncomb(x,y; Δx, Δy) = Σm,n δ ( x - mΔx, y - nΔy ) ~ separable functionFT: COMB(ζx, ζy) = comb(ζx, ζy; 1/Δx, 1/Δy) / ΔxΔy

Sampling vs. Replication (tiling) – Nyquist rates (2ζx0 and 2ζy0) − Aliasing

Jain’s Fig.4.7

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01)


22--D Sampling: Beyond Rectangular GridD Sampling: Beyond Rectangular GridSampling at nonrectangular grid – May give more efficient sampling

density when spectrum region of support is not rectangular

Sampling density measured by #samples needed per unit area

– E.g. interlaced grid for diamond-shaped region of support

equiv. to rotate 45-deg. of rectangular gridspectrum rotate by thesame degree

From Wang’s book preprint Fig.4.2

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01/2

004)

33


General Sampling LatticeGeneral Sampling LatticeLattice Λ in K-dimension space R K

– A set of all possible vectors represented as integer weighted combinations of K linearly independent basis vectors

Generating matrix V (sampling matrix)V = [v1, v2, …, vk] => lattice points x = V n

e.g., identity matrix V ~ square lattice

Voronoi cell of a lattice– A “unit cell” of a lattice, whose translations cover the whole space– Consists of vectors that are closer to the origin than to other lattice points

cell boundaries are equidistant lines between surrounding lattice points

Sampling density d(Λ) = 1 / |det(V)|– |det(V)| measures volume of a cell; d(Λ) is # lattice points in unit volume

⎭⎬⎫

⎩⎨⎧

∈∀=∈=Λ ∑=

K

jkjj

K nn1

,| ZR vxx

From Wang’s book preprint Fig.3.1U

MC

P EN

EE63

1 Sl

ides

(cre

ated

by

M.W

u ©

2001

/200

4)


Sampling Density:d1 = 1d2 = 2 / √3

)(hexagonal 12/102/3

ar)(rectangul 1001

2

1

⎥⎦

⎤⎢⎣

⎡=

⎥⎦

⎤⎢⎣

⎡=

V

V


Example of LatticesExample of Lattices

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

04)


Frequency Domain View & Reciprocal LatticeFrequency Domain View & Reciprocal LatticeReciprocal lattice Λ# for a lattice Λ (with generating matrix V)– Generating matrix of Λ# is U = (VT)-1

– Basis vectors for Λ and Λ# are orthonormal to each other: VT U = I– Denser lattice Λ has sparser reciprocal lattice Λ# : det(U) = 1 / det(V)

Frequency domain view of sampling over lattice– Sampling in spatial domain Repetition in freq. Domain– Repetition grid in freq. domain can be described by reciprocal lattice

Aliasing and prefiltering to avoid aliasing– Aliasing happens when Voronoi cell of reciprocal lattice overlapped

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01/2

004)


Sampling EfficiencySampling EfficiencyConsider spherical signal spectrum support– Most real-world signals have symmetric freq. contents in many directions– The multi-dim spectrum can be approximated well by a sphere (with proper

scaling spectrum support)

Voronoi cell of reciprocal lattice need to cover the sphere to avoid aliasing– Tighter fit of the Voronoi cell to the sphere requires less sampling density

What lattice gives the best sphere-covering capability?Sampling Efficiency ρ = volume(unit sphere) / d(Λ) prefer close to 1

From Wang’s book preprint Fig.4.2 & 3.5

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01/2

004)

44


Sampling Lattice ConversionSampling Lattice ConversionFrom Wang’s book

preprint Fig.4.4

Intermediate

Original

TargetedUMCP ENEE631 Slides (created by M.Wu © 2001)


Recall: 1Recall: 1--D D UpsampleUpsample and and DownsampleDownsample

From Crochiere-Rabiner “Multirate DSP” book Fig.2.15-16

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01)


General Procedures for Sampling Rate ConversionGeneral Procedures for Sampling Rate Conversion


UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01)


Example: Frame Rate ConversionExample: Frame Rate ConversionVideo sampling: formulate as a 3-D sampling problemNote: different signal characteristics and visual sensitivities along spatial

and temporal dimensions (see Wang’s Sec.3.3 on video sampling)

General Approach to frame rate conversion– Upsample => LPF => Downsample

Interlaced 50 fields/sec 60 fields/sec– Analyze in terms of 2-D sampling lattice (y, t) – Convert odd field rate and even field rate separately

do 25 30 rate conversion twicenot fully utilize info. in the other fields

– Deinterlace first then convert frame ratedo 50 60 frame rate conversion: 50 300 60

– Simplify 50 60 by converting 5 frames 6 frameseach of output 6 frames is from two nearest frames of the 5 originalsweights are inversely proportional to the distance between I/O

– May do motion-interpolation for hybrid-coded video

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01/2

004)

55



UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01)


Case Studies on Sampling and Case Studies on Sampling and ResamplingResamplingin Video Processingin Video Processing

Reading Assignment: WangReading Assignment: Wang’’s book Chapter 4s book Chapter 4

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

04)


Video Format Conversion for NTSC Video Format Conversion for NTSC PALPAL

Require both temporal and spatial rate conversion– NTSC 525 lines per picture, 60 fields per second– PAL 625 lines per picture, 50 fields per second

Ideal approach (direct conversion)– 525 lines 60 field/sec 13125 line 300 field/sec

625 lines 50 field/sec

4-step sequential conversion– Deinterlace => line rate conversion

=> frame rate conversion => interlace

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01)



UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01)

66


Simplified Video Format ConversionSimplified Video Format Conversion

50 field/sec 60 field/sec– Simplified after deinterlacing to 5 frames 6 frames– Conversion involves two adjacent frames only

625 lines 525 lines– Simplified to 25 lines 21 lines– Conversion involves two adjacent lines only

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01)


UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01)


Interlaced Video and Interlaced Video and DeinterlacingDeinterlacingInterlaced videoOdd field at 0 Even field at Δt Odd field at 2Δt Even field at 3Δt

…

Deinterlacing– Merge to get a complete frame with odd and even field

Examples from http://www.geocities.com/lukesvideo/interlacing.html

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01)


DeDe--interlacing: Practical Approachesinterlacing: Practical Approaches

Spatial interpolation– Vertical interpolation within

the same field (1-D upsample by 2)– Line averaging ~ average the line

above and below D=(C+E)/2

Temporal interpolation– 2-frame field merging => artifacts– 3-frame field averaging D=(K+R)/2

fill in the missing odd field by averaging odd fields before and after

Spatial-temporal interpolation– Line-and-field averaging D=(C+E+K+R)/4

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01)

77


MotionMotion--Compensated DeCompensated De--interlacing interlacing

Stationary video scenes– Temporary deinterlacing approach yield good result

Scenes with rapid temporal changes– Artifacts incurred from temporal interpolation– Spatial interpolation alone is better than involving temporal

interpolation

Switching between spatial & temporal interpolation modes– Based on motion detection result– Hard switching or weighted average– Motion-compensated interpolation

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

01)


Summary of TodaySummary of Today’’s Lectures LectureSampling and resampling issues in 2-D and 3-D– Sampling lattice and frequency-domain interpretation– Sampling rate conversion

Next Lecture: – Introduction to digital watermarking for image and video

Readings– Wang’s book: Sec. 3.1-3.3, 3.5; Chapter 4– “Computer Graphics” Chapter 5 (Hearn-Baker, Prentice-

Hall, 2nd Ed)

UM

CP

ENEE

631

Slid

es (c

reat

ed b

y M

.Wu

©20

04)

sampling issues in image and video z – 2-d sampling at ... · pdf filesampling issues in...

Documents