fourier analysis of video signals
TRANSCRIPT
-
8/12/2019 Fourier Analysis of Video Signals
1/29
Fourier Analysis of Video Signals
Frequency Response of the HVS
Yao Wang
Polytechnic University, Brooklyn, [email protected]
-
8/12/2019 Fourier Analysis of Video Signals
2/29
Yao Wang, 2003 Frequency Domain Analysis
Outline
Fourier transform over multidimensional space Continuous space FT (CSFT) Discrete space FT (DSFT) Sampled space FT (SSFT)
Frequency domain characterization of video signals
Spatial frequency Temporal frequency Temporal frequency caused by motion
Frequency response of the HVS
Spatial frequency response Temporal frequency response and flicker Spatio-temporal response Smooth pursuit eye movement
Video sampling a brief discussion
-
8/12/2019 Fourier Analysis of Video Signals
3/29
Yao Wang, 2003 Frequency Domain Analysis 3
Continuous Space Signals
K-D Space Signals
Convolution
Example function Delta function
k K R x x x = ],...,,[),( 21xx
yyyxxx d hhk R = )()()(*)(
-
8/12/2019 Fourier Analysis of Video Signals
4/29
Yao Wang, 2003 Frequency Domain Analysis 4
Continuous Space Fourier Transform
(CSFT) Forward transform
Inverse transform
Convolution theorem
xxf xf d jk R
T c = )2exp()()(
f xf f x d jk R
T c= )2exp()()(
)(*)()()(
)()()(*)(
f f xx
f f xx
cc
cc
H h
H h
-
8/12/2019 Fourier Analysis of Video Signals
5/29
Yao Wang, 2003 Frequency Domain Analysis 5
Continuous Space Systems
General system over K-D continuous space
Linear and Space-Shift Invariant (LSI) System
LSI systems can be completely described by itsimpulse response
k RT = xxx )),(()(
( ))()()()(*)()(
)()(
f f f xxx
xx
ccc H h
T h==
=
)())((
))()(()()(
00
22112211
xxxx
xxxx+=+
+=+
T
T
-
8/12/2019 Fourier Analysis of Video Signals
6/29 Yao Wang, 2003 Frequency Domain Analysis 6
Discrete Space Signals
K-D Space Signals
Convolution
Example function Delta function
K K Z nnn = ],...,,[),( 21nn
= K Z
hhm
mmnnn )()()(*)(
-
8/12/2019 Fourier Analysis of Video Signals
7/29 Yao Wang, 2003 Frequency Domain Analysis 7
Discrete Space Fourier Transform
(DSFT) Forward transform
Inverse transform
Convolution theorem
{ })2/1,2/1(, : periodlFundamenta1of periodwithdimensioneachin periodicis)(
)2exp()()(
=
=
k K
d
R
T d
f I
j K
f
f
nf nf n
= K
I
T d d j
f
f nf f n )2exp()()(
)(*)()()(
)()()(*)(
f f nn
f f nn
d d
d d
H h
H h
-
8/12/2019 Fourier Analysis of Video Signals
8/29 Yao Wang, 2003 Frequency Domain Analysis
Frequency domain characterization of
video signals Spatial frequency
Temporal frequency Temporal frequency caused by motion
-
8/12/2019 Fourier Analysis of Video Signals
9/29 Yao Wang, 2003 Frequency Domain Analysis 9
Spatial Frequency
Spatial frequency measures how fast the image
intensity changes in the image plane Spatial frequency can be completely characterized by
the variation frequencies in two orthogonal directions
(e.g horizontal and vertical) f x : cycles/horizontal unit distance f y : cycles/vertical unit distance
It can also be specified by magnitude and angle ofchange
)/arctan(,22 x y y xm f f f f f =+=
-
8/12/2019 Fourier Analysis of Video Signals
10/29 Yao Wang, 2003 Frequency Domain Analysis 10
Illustration of Spatial Frequency
-
8/12/2019 Fourier Analysis of Video Signals
11/29 Yao Wang, 2003 Frequency Domain Analysis 11
Angular Frequency
ee)cycle/degr (f 180
f f
(degree)180
n)h/2d(radia(radian))2/arctan(2
s s
hd
d h
d h
==
==
Problem with previous defined spatial frequency:Perceived speed of change depends on the viewing distance.
-
8/12/2019 Fourier Analysis of Video Signals
12/29 Yao Wang, 2003 Frequency Domain Analysis 12
Temporal Frequency
Temporal frequency measures temporal variation
(cycles/s) In a video, the temporal frequency is spatial position
dependent, as every point may change differently
Temporal frequency is caused by camera or objectmotion
-
8/12/2019 Fourier Analysis of Video Signals
13/29 Yao Wang, 2003 Frequency Domain Analysis 13
Temporal Frequency
caused by Linear Motion
-
8/12/2019 Fourier Analysis of Video Signals
14/29 Yao Wang, 2003 Frequency Domain Analysis 14
)(
:frequencytemporalandspatial,motion, betweenRelation
)(),(),,(
),(),,(
0
0
y y x xt
y y x xt y xt y x
y x
f v f v f
f v f v f f f f f f
t v yt v xt y x
+=
++=
=
The temporal frequency of the image of a moving object depends on motion aswell as the spatial frequency of the object.
Example: A plane with vertical bar pattern, moving vertically, causes notemporal change; But moving horizontally, it causes fastest temporal change
Relation between Motion, Spatial and
Temporal Frequency
isat time patternimage the),,(is
0at patternimagetheAssume ).,(speedwithmovingobjectanConsider
0 t y x
t vv y x
=
-
8/12/2019 Fourier Analysis of Video Signals
15/29 Yao Wang, 2003 Frequency Domain Analysis 15
Illustration of the Relation
-
8/12/2019 Fourier Analysis of Video Signals
16/29
-
8/12/2019 Fourier Analysis of Video Signals
17/29 Yao Wang, 2003 Frequency Domain Analysis 17
Temporal Response
200
100
50
20
10
5
2
12 5 10 20 50
Frequency (Hz)
C o n
t r a s t s e n s i
t i v
i t y
0.06 trolands
850 trolands9300 trolands
7.1 trolands77 trolands
0.65 trolands
Figure 2.5 The temporal frequency response of the HVS obtained by a visualexperiment. Different curves represent the responses obtained with different meanbrightness levels, B, measured in trolands. The horizontal axis represents the ickerfrequency f , measured in Hz. Reprinted from D. H. Kelly, Visual responses totime-dependent stimuli. I. Amplitude sensitivity measurements, J. Opt. Soc. Am.(1961) 51:42229, by permission of the Optical Society of America.
Critical flicker frequency: The
lowest frame rate at which the eyedoes not perceive flicker.
Provides guideline for determiningthe frame rate when designing a
video system.
Critical flicker frequency dependson the mean brightness of thedisplay:
60 Hz is typically sufficient forwatching TV.
Watching a movie needs lower
frame rate than TV
-
8/12/2019 Fourier Analysis of Video Signals
18/29 Yao Wang, 2003 Frequency Domain Analysis 18
Spatial Response
100
50
C o n
t r a s t s e n s i
t i v
i t y
20
10
5
2
0.21
0.5 1Spatial frequency (cpd)
2 5 10
Figure 2.6 The spatial frequencyresponse of the HVS, obtained by a visualexperiment. The three curves result fromdifferent stabilization settings used toremove the effect of saccadic eyemovements. Filled circles were obtainedunder normal, unstablized conditions;open squares, with optimal gain settingfor stabilization; open circles, with thegain changed about 5 percent. Reprintedfrom D. H. Kelly, Motion and vision. I.Stabilized images of stationary gratings, J. Opt. Soc. Am. (1979), 69:126674, bypermission of the Optical Society of America.
-
8/12/2019 Fourier Analysis of Video Signals
19/29 Yao Wang, 2003 Frequency Domain Analysis 19
Spatiotemporal Response
0.3
3
1 3 10
Spatial frequency (cpd)
(a)
30
10
30
100
300
C
o n
t r a s t s e n s i
t i v
i t y
0.3
3
1 3 10
Temporal frequency (Hz)
(b)
30
10
30
100
300
C
o n
t r a s t s e n s i
t i v
i t y
Figure 2.7 Spatiotemporal frequency response of the HVS. (a) Spatial frequency responsesfor different temporal frequencies of 1 Hz (open circles), 6 Hz (lled circles), 16 Hz (opentriangles), and 22 Hz (lled triangles). (b) Temporal frequency responses for different spatialfrequencies of 0.5 cpd (open circles), 4 cpd (lled circles), 16 cpd (open triangles), and 22 cpd(lled triangles). Reprinted from J. G. Robson, Spatial and temporal contrast sensitivityfunctions of the visual systems, J. Opt. Soc. Am. (1966), 56:114142, by permission of theOptical Society of America.
The reciprocal relationbetween spatial and
temporal sensitivity wasused in TV system design:
Interlaced scan providestradeoff between spatialand temporal resolution.
-
8/12/2019 Fourier Analysis of Video Signals
20/29 Yao Wang, 2003 Frequency Domain Analysis 20
Smooth Pursuit Eye Movement
Smooth Pursuit: the eye tracks moving objects
Net effect: reduce the velocity of moving objects on the retinal plane,so that the eye can perceive much higher raw temporal frequenciesthan indicated by the temporal frequency response.
y y x xt
y y x xt t
y x
y y x xt
y x
vvvv f
f v f v f f
vv
f v f v f
vv
===
++=
+=
~,~ if 0~
)~~(~
:)~,~(atmovingiseye when theretinaat thefrequencytemporalObserved
)(
:),(atmovingisobjectn themotion wheobject bycausedfrequencyTemporal
-
8/12/2019 Fourier Analysis of Video Signals
21/29
(b)
S p a t i a l f r e q u e n c y ( c p d ) T
e m p o
r a l f r e q u e
n c y ( H z
) C
o n
t r a s t s e n s i
t i v i t y
0.15
0
010
20
30
510
1520
0.1
0
0.05
(a)
S p a t i a l f r e q u e n c y ( c p d )
0.15
0.1
0
0.05
0
010
20
30
510
15
20 T e m p o
r a l f r e q u e n c y ( H z
)
(c)
S p a t i a l f r e q u e n c y ( c p d )
0.15
0.1
0
0.05
0
010
20
30
510
15
20 T e m p o
r a l f r e q u e n c y ( H z
)
Figure 2.8 Spatiotemporal response of the HVS under smooth pursuit eye movements:(a) without smooth pursuit eye movement; (b) with eye velocity of 2 deg/s; (c) with eyevelocity of 10 deg/s. Reprinted from Girod, B. Motion compensation: visual aspects, accuracy,and fundamental limits. In Sezan, M. I., and R. L. Lagendijk, eds., Motion Analysis and ImageSequence Processing, Boston: Kluwer Academic Publishers, 1993, 12652, by permission of Kluwer Academic Publishers.
-
8/12/2019 Fourier Analysis of Video Signals
22/29
Yao Wang, 2003 Frequency Domain Analysis
Video Sampling A Brief Discussion
Review of Nyquist sampling theorem in 1-D
Extension to multi-dimensions Prefilter in video cameras Interpolation filter in video displays
-
8/12/2019 Fourier Analysis of Video Signals
23/29
Yao Wang, 2003 Frequency Domain Analysis 23
Nyquist Sampling Theorem in 1-D
Given a band-limited signal with maximum frequency f max, it canbe sampled with a sampling rate f
s>=2 f
max. The original
continuous signal can be reconstructed (interpolated) from thesamples exactly, by using an ideal low pass filter with cut-offfrequency at f s /2.
Practical interpolation filters: replication (sample-and-hold, 0th
order), linear interpolation (1 st order), cubic-spline (2 nd order) Given the maximally feasible sampling rate f s, the original signal
should be bandlimited to f s /2, to avoid aliasing. The desired
prefilter is an ideal low-pass filter with cut-off frequency at f s /2. Prefilter design: Trade-off between aliasing and loss of high
frequency
-
8/12/2019 Fourier Analysis of Video Signals
24/29
Yao Wang, 2003 Frequency Domain Analysis 24
Extension to Multi-dimensions
If the sampling grid is aligned in each dimension (rectangular in2-D) and one performs sampling in each dimension separately,the extension is straightforward: Requirement: f s,i
-
8/12/2019 Fourier Analysis of Video Signals
25/29
-
8/12/2019 Fourier Analysis of Video Signals
26/29
Yao Wang, 2003 Frequency Domain Analysis 26
Video Cameras
Sampling mechanism All perform sampling in time Film cameras capture continuous frames on film Analog video cameras sample in vertical but not horizonal
direction, arrange the resulting horizontal scan lines in a 1-Dcontinuous signal
Digital cameras sample in both horizontal and vertical direction,yielding pixels with discrete 3-D coordinates
Sampling frequency (frame rate and line rate) Depending on the maximum frequency in the underlying signal, the
human visual thresholds, as well as technical feasibility and cost Prefilter
Controlled by temporal exposure, scanning beam, etc. Digital cameras may capture at higher sampling rates and then
implement explicit filtering before converting to lower resolution
-
8/12/2019 Fourier Analysis of Video Signals
27/29
Yao Wang, 2003 Frequency Domain Analysis 27
Typical Camera Response
Temporal prefilter: the value read out at any frame is
the average of the sensed signal over the exposuretime
Spatial prefilter: the value read out at any pixel is aweighted integration of the signal in a small windowsurrounding it, called the aperture, can be
approximated by a box average or a 2-D Gaussianfunction
-
8/12/2019 Fourier Analysis of Video Signals
28/29
Yao Wang, 2003 Frequency Domain Analysis 28
Video Display
The display device presents the analog or digital video on thescreen to create the sensation of continuously varying signal inboth time and space.
With CRT, three electronic beams strike red, green, and bluephosphors with the desired intensity at each pixel location. No
explicit interpolation filters are used. Spatial filtering determinedby the size of the scanning beam, temporal filtering determinedby the decaying time of the phosphors.
The eye performs the interpolation task: fuses discrete frames
and pixels as continuously varying, if the temporal and spatialsampling rates are sufficiently high.
-
8/12/2019 Fourier Analysis of Video Signals
29/29
Homework 2
Reading assignment:
Chapter 2 Chapter 3, Section 3.4
Written assignment
Prob. 2.1,2.2,2.5,2.6,2.7