advanced transform methods
TRANSCRIPT
Advanced Transform MethodsProfessor Sir Michael Brady FRS FREng
Department of Engineering ScienceOxford University
Hilary Term 2006
Outline
• Recap on Fourier domain• Limitations of the Fourier transform• Gabor’s great idea• Analytic signal, local phase & features• Bases for spaces of functions• Wavelets• The Riesz transform, monogenic signal &
features again
Recap on Fourier domain
• Frequency for signal/image processing– “Frequency freaks” vs “feature creatures”
• Fourier’s idea– Was using sine/cosine lucky or insightful?
• Efficiency: the FFT• The importance of phase• Homomorphic filtering• Image enhancement & restoration
Frequency for signal/image analysis
+ “noise” is often high frequency– Unfortunately, so are transients such as edges, corners, …– The human visual system ignores constant signals but detects
signal changes– Filtering out high frequencies can damage edges
+ Textures often have repeated patterns that correspond to peaks in Fourier spectrum– Unfortunately, this is too simple for texture analysis in practice
+ Frequency codes for position in MRI+ Sadly, Fourier theory is often ignored in image analysis –
at cost of poorer results than necessary– Example of rotational invariance of feature detection
Frequency freaks
Sinusoidal grating
Campbell and Robson’s contrast sensitivity function CSF
Example of a CSF: threshold contrast vs spatial frequency (cycles/degree or lp/mm)
Output from the monogenic signal
a. Corners detected by a conventional algorithm as a shape rotates;
c. Edges detected by the Canny edge detector as the shape rotates
b. Corners detected after Fourier analysis of the filter response is analysed as a function of orientation, leading to a filter whose response is anisotropic
d. Edges after similar analysis
Source: DPhil thesis by Jason Merron, 1997
Fourier’s idea (1807)Fourier presented a memoir to the Institut de France in which he claimed that all continuous functions of a real variable f(t) can be represented as series of harmonically related sines and cosines. The complete mathematical story was not worked out until 1960!
Definition: Let f(t) be a continuous function of a real variable t, then the Fourier Transform of f is defined to be:
∫∞
∞−
−→ℑ dtetfuf jut)( :)(
ℑNote that is an operator that transforms a function f of one variable into another:
)()(: 11 ℜ→ℜℑ LL∫ +∞<ℜ dtL f(t) :is that functions, integrable of space (Hilbert) theis )( where 1
Inverse FT usually exists
An important result is that for most functions f and for most t the inverse transform exists too:
ff ˆby )( denote :otation ℑN
∫∞
∞−
=
ℜ∈ℜ∈
dueuftf
tLf
jut)(ˆ21)(
, allalmost for then ,)( If 1
π
Power and phase
)(ˆ)(ˆ
)(tan
)(ˆ)(ˆ)(ˆ
where
)(ˆ)(ˆ :form lexponentiain Express
)(ˆ)(ˆ)(ˆfunction luedcomplex va a is )(ˆ that Note
22
)(
ufufu
ufufuf
eufuf
ufjufufuf
r
i
ir
uj
ir
=
+=
=
+=
φ
φ
The first of these is the square root of the power spectrum of f, the second is the phase angle of f.
Most of signal processing is concerned with the power spectrum; but (as we shall see) phase dominates in image analysis
Decomposing an image into its spatial frequency components
Rectangle function and sinc
)(sinc)(
)sin()(
21||,021||,1
)(
XuXXu
XuXuF
x
xxf
πππ
==
⎪⎩
⎪⎨
⎧
≥
<=
Rectangle function plays a key role in image/signal processing. In the frequency domain it defines a perfect bandpass filter, in the space domain a perfect windowing operator.
Sinc interpolation plays an important role in MRI image analysis, and geometric alignment of data sets.
Fourier transform properties
)(ˆ)(derivativeFrequency )(ˆ)()(derivative Time
)(ˆ)( Scaling
)(ˆModulation)(ˆ)(nTranslatio
)(ˆ)(ˆ21))((nConvolutio
)(2)(ˆInverse)(ˆ)(property
0
00
0
ufjtufjutf
sufsstf
uufeufettf
ufuhtfh
uftfuftf
pp
pp
tju
ujt
−
−−
∗
−
−π
π
Examples, and a question
( )
1)(2
sinc)(
)()(21cos
t
t
t
δ
ω
λωδλωδλ
Π
−++
A reminder from the FT course:
OK, so “all continuous functions of a real variable f(t) can be represented as series of harmonically related sines and cosines”
But was Fourier's observation just lucky? Or, is there a deeperreason why cosines and sines - complex exponentials - should play a key role?
Linear, time invariant filtering
βαβαβα
, constantsany for )()()( iflinear isoperator An gfgf LLL L +=+
Linearity is a simplifying assumption that is made throughout much of engineering science, even if it rarely holds exactly.
))(( ))(( then),((t) if is,That .by delayed also isoutput then the
, timeaby delayed is input to theifinvariant timecalled is operator An
tftftff
f
τ
τ
τττ
τ
LL
L
=−−=
linear, time-invariant operators: impulses and convolution
)()( denote and1011
)(
function Dirac the)(by Denote
utttt
t
t
u −=⎩⎨⎧
≠=
=
δδ
δ
δ
Linear, time-invariant systems are characterised by their response toDirac functions. Assume for the moment that the followingintegrations are defined: ∫
∞
∞−
= dttuftf u )()()( δ
dttuftf u )()()( impliesLinearity δLL ∫∞
∞−
=
Impulse response
The impulse response h of the linear, time-invariant operator L is
)()( tth δL=
))((
)()(
)()()(
thatso ),()( that proves invariance time
tfh
duutfuh
duuthuftf
uthtu
∗=
−=
−=
−=
∫
∫∞
∞−
∞
∞−
L
Lδ
The impulse response function of a linear time-invariant operator is a convolution
complex exponentials are eigenfunctions of convolution operators
ωfrequency at the of FT theis )()(ˆ eigenvalue The
)(ˆ
)(
)( )(
hdueuhwh
ewh
dueuhe
dueuhe
jwu
jwt
jwujwt
utjwjwt
∫
∫
∫
∞
∞−
∞
∞−
∞
∞−
−
=
=
=
=L
… so linearity impulse response convolution sines & cosines
So Fourier's insight was deep, though it may also have been lucky!
Convolution theorem and FFTWe have seen that linear, time-invariant operators correspond precisely with those defined by convolution. That is, L is a linear, time-invariant operator if, and only if, there is a function h(t), called the impulse-response function of L so that Lf = h*f.
The convolution theorem is the core of the application of Fourier theory to signal and image processing, since the naïve shift-and-multiply implementation of convolution is intrinsically expensive, havingcomplexity O(N2) where N is the desired number of values of u.
( ) fhfh
LfhLhLf
ˆˆ*
and )( then ,)( and )( If :Theoremn Convolutio^
111
=
ℜ∈∗ℜ∈ℜ∈
Convolution in the time/space domain = multiplication in Fourier domain
Practical importance of the convolution theorem
),(),(),(),( vuGvuFyxgyxf ⇔∗
Smoothing an image with a Gaussian kernel. Could do it in the space domain, using separability of the Gaussian, OR1. Compute the FT of the image and of the
Gaussian kernel
2. Multiply their FTs
3. Compute inverse FT of productThis only makes sense because there is a FAST FT algorithm
The FFT
∑∑−
=
−
=
−==
1
0
1
0)(1)(1)(ˆ
N
t
utN
N
t
Njut
WtfN
etfN
tf
Assume N = 2n for some n, so that N = 2M for some M. Substituting:
⎥⎦
⎤⎢⎣
⎡++= ∑∑
−
=
−
=
uM
M
t
utM
M
t
utM WWtf
MWtf
Muf 2
1
0
1
0)12(1)2(1
21)(ˆ
So that:
[ ]uMWufufuf 2oddeven )(ˆ)(ˆ
21)(ˆ +=
FFT algorithmGiven a signal {f(0), f(1), …., f(63)}, it is broken down in to:
• {f(0), f(2), … , f(62)} and
• {f(1), f(3), …, f(63)
That is, the FT of a signal of length 64 can be computed from two FTs of length 32. These in turn are broken down in to their odd and even bits
{f(0), f(4), …}; {f(2), f(6), …}; {f(1), f(5), …}; {f(3), f(7),..}
So it can be computed from 4 of length 16. In general, For a signal of length N, the algorithm takes time N log N instead of N2 by the naïve method.
There is nothing magical about this process, it follows from simple properties of sines and cosines. Exactly, the same idea works for wavelets
2D FT
∫ ∫
∫ ∫∞
∞−
∞
∞−
+
∞
∞−
∞
∞−
+−
=
=
dudvevufyxf
dxdyeyxfvuf
vufyxf
vyuxj
vyuxj
)(
)(
),(ˆ),(
),(),(ˆ
definecan then we ,integrable is ),(ˆ if and ,integrable and continuous is ),( If
We can define the power and phase as for 1D FT
FT pair in 2Dv
)()(),( YvsncXuXYsncvuF ππ=
FT of a circular disk
ρρπρ /)()(),(||,0||,1
),(
1 aaJFvuFarar
yxf
==⎩⎨⎧
≥<
=
2D version of a sinc function
Important in ultrasound image analysis
2D FT of an X shape
Discrete FT
∑
∑
∑
∞
−∞=
∞
−∞=
−
∞
−∞=
−=
=
−=
±
nd
n
jnuTd
nd
Tnuf
Tuf
Tf
enTfuf
nTtnTftf
nTfnTtftf
)2(ˆ1)(ˆ
:is intervalsat uniformly samplingby obtained signal discrete theof FT The Theorem
)()(ˆ
:find we1, is delta a of FT thefact that theand ,on theorem translati theUsing
)()()(
:sum Dirac weighteda toscorrespond )( sample discreteA ).( into sampled is )(function The 0
π
δ
This shows that sampling f at intervals T is equivalent to making its Fourier transform periodic by summing all its translations.
Discrete image filteringThe properties of 2D space-invariant operators are essentially the same as in 1D. A linear operator L is space-invariant if, for any
),(),(
then),,(),(
,
,
qmpnfmnf
qmpnfmnf
qp
qp
−−=
−−=
LL
An image can be decomposed as a sum of discrete Diracs and then linearity and space-invariance proves that any L has an associated impulse response function h(n,m) so that application of L is equivalent to a 2D convolution with h
SeparabilityThe impulse response function h(n,m) is separable if it is a product of two one-dimensional functions:
)()(),( 21 mhnhmnh =In the case of a separable impulse response function, the two-dimensional convolution can be rewritten as a 1D convolution along the columns followed by a 1D convolution along the rows (or viceversa).
( ) ∑ ∑∞
−∞=
∞
−∞=⎟⎟⎠
⎞⎜⎜⎝
⎛−−=∗
p qqpfqmhpnhmnhf ),()()(),( 21
Reduces compute complexity to O(2N); the discrete FT can be computed by column then by row.Example: Gaussian convolution and pipeline architectures
FT of a Gaussian
2222
2222/2
,)(),(
,2
1)(222
22
vueFvuF
yxrerf r
+===
+==
−
−
ρρπσ
σρπ
σ
• FT of a Gaussian is a Gaussian
• Note the inverse scale relation
Power spectrum of FT of an image
f(x,y)
|F(u,v)|
original Low pass High pass
Image with periodic structure
f(x,y) |F(u,v)|
FT has peaks at spatial frequencies of repeated textures
Forensic application
|F(u,v)|
Remove peaks, equals periodic background
Energy and Phase
Seems like phase carries less information about the image …
The importance of phaseOppenheim’s demonstration
I1 I2
FFT FFT
FFT-1
I3
Do you see: I1, I2, neither, or both?
),(11
1),(),(ˆ vujevuPvuI Φ= ),(22
2),(),(ˆ vujevuPvuI Φ=
),(13
2),(),(ˆ vujevuPvuI Φ=
Keble Another dinosaur
Keble phases the dinosaur
Keble Another dinosaur
the dinosaur phases Keble
Step edge detection by differentiation
inte
nsity
Signal position, x
Profile of a step intensity change superimposed on a ramp
Basic feature detection -conventional approach
inte
nsity
Signal position, x
Profile of a step intensity change superimposed on a ramp
d in
tens
ity/ d
x
Constant gradient + max at the step. Note amplified noise
Differentiation by the central difference is equivalent to convolution with the odd symmetric kernel
Differentiation as convolution
2)1()1()( −−+
≈=
ififdx
xdf
ix
Approximate differentiation as central difference:
⎥⎦⎤
⎢⎣⎡−
21,0,
21
Problem is that differentiation amplifies noise, suggesting pre-filtering the image with a noise reducing, smoothing filter
Convolution with the derivative of a Gaussian
)()( xfdx
xdG∗σ
Step detection by finding “significant” maxima of the convolution of the signal with a derivative of a Gaussian
Step superimposed on a ramp signal… take second derivative
Gaussian derivatives in 2D
),(21
2
),(2
),(),(
2222
222
2/2/3
2/)(4
yxfeex
yxfexyxfyxGx
yx
yx
∗⎟⎠
⎞⎜⎝
⎛∗⎟⎠
⎞⎜⎝
⎛ −=
∗⎟⎠⎞
⎜⎝⎛ −
=∗∂∂
−−
+−
σσ
σ
σπσπ
πσGaussian is separable: equivalent to convolution in the x-direction and then in the y-direction
Directional derivative of Gaussian shown as a surface
Example
X-derivative Y-derivative
Gradient magnitude
Basic feature detection –zero crossingsin
tens
ity
Signal position, x
Profile of a step intensity change superimposed on a ramp
d2in
tens
ity/ d
x2
Zero crossing at step. But doubly amplified noise
Gaussian scale space
This shows a Gaussian blurred signal Gσ*S(t) as σ increases (from bottom to top)
The positions of the zero-crossings of the second derivative of the blurred signal are marked. This produces a “fingerprint”
Homomorphic filteringIllumination model: i(x,y)= illum(x,y)reflec(x,y) where i(x,y)is the image, illum(x,y) is the spatially slowly-varying illumination, and reflec(x,y) is the reflectance of visible surfaces, assumed piecewise constant.
Taking logs to convert to a sum (indicated by Capitalised names),then apply the FT:
)),(),(()),(),((),(
:FT inverse then takeand ),(filter suitable aApply ),(),(),(ˆ
^1
^1
^
^^
vuReflecvuHvuIllumvuHyxEnh
vuHvuReflecvuIllumvuI
−− ℑ+ℑ=
+=
Then the enhanced image is obtained by taking exponents of this equation.
Homomorphic filtering: example
Left: Original image. Right: processed using homomorphic filteringto achieve contrast enhancement and dynamic range compression.
Homomorphic filtering in practice
• The choice of the filter H(u,v) is critical– it must affect low and high frequencies in different
ways – ``illumination varies spatially slowly'' suggests low-
pass– H(u,v) is typically ``1-Butterworth''
• Note that noise was ignored …• Worse, reflectance also has a low frequency
component, so separation is often hard– ``bias field'' in MRI image enhancement
Signal/image restoration using the Fourier domain
The objective is to restore a degraded image to its original form. An observed image can often be modelled as:
),(),(),(),( yxnyxfyxhyxg +∗=
where f is the signal/image to be restored, h is the point spread function of the imaging process, n is additive noise, and g is the signal/image with which we are confronted. The aim is to restore f given g and some knowledge of n and h. Evidently: ),(ˆ),(ˆ),(ˆ),(ˆ vunvufvuhvug +=
ˆ~
-ˆˆ~ :noise gintroducin-re or,ˆ
ˆ~
:bygiven is ˆ of ~ estimatean moment, for the ˆ Ignoring
hn
hgf
hgf
ffn
==
Evidently, problems if or is small, so noise dominates0ˆ ≈h g
Wiener filter
signal theofdensity spectralpower theis ),(~),(
noise theofdensity spectralpower theis),(ˆ ),(
),(/),(),(
where),(),(ˆ
),(ˆ),(ˆ
where),,(ˆ),(ˆ),(~
2
2
2
*
vufvuS
vunvuS
vuSvuSvuk
vukvuhvuhvuw
vuwvugvuf
f
f
=
=
=
+=
=
η
η
constant suitable a toequalset is or ely,approximatknown are ,Often
attenuated are sfrequenciehigh then , largefor ),(ˆ),( if
filter) inverse (ie ),(ˆ/1),(~ then ,0),( If
kSS
vuvuhvuk
vuhvufvuk
f η
>>
==
Wiener filter example
Here, K is set to the constant value indicated
Inadequacies of the FT & attempts to overcome them
• The discrete world, signal changes, localisation
• Localisation vs SNR• Gaussian “scale space”• Isotropic and anisotropic diffusion for
image analysis• Estimating orientation in images :
steerable filters
The transient world
A sine function with a small discontinuity.The Fourier Transform consists of two delta functions, indicating the single dominant signal. Note the infinite tiny wiggles –across the entire spectrum.It detects that there is a discontinuity; but cannot say where
The basic problem with the FT for signal/image processing
Using the FT of the Rect function, we see that the FT of any function that is bounded in time, has frequencies over an infinite range. Conversely, the inverse FT of a perfect band-pass filter has temporal components over an infinite range.
We may detect that a transient has occurred; but we cannot detect when! If we are to treat transients in signal/image processing, we must find a way to overcome this problem. (The great idea in MRI is to add a gradient field to provide localisation; but that is infeasible for most signal and image processing.)
upperlower tttttf ≥≤= or for ,0)(
Space/time & frequency
• Shannon (1943) • Fourier (1807)
– the FFT– Human visual system “Frequency freaks” vs “feature
creatures”• Gabor (1950)
– The Heisenberg map• Wavelets (1978)
– the transient world
Heisenberg plane
t
ω
This is a visual representation which aims to convey the way in which sampling is done between time (space) and frequency/scale
Shannon or time-sampled representation
t
ω
The signal is sampled in the time domain, and the FT of each time sample covers all frequencies
The diagram attempts to capture the idea that Shannon/Nyquist theory samples signals in time; but that a rectangle function in time has infinite extent in frequency
Representation of the Fourier Transform
t
ω The signal is sampled in the frequency domain, and the inverse FT of each frequency sample is spread over all times
The practical import of the Fourier Transform* is that global structure of a signal (eg a sinusoid) is converted to a single localised point.
*This is the key property of ALL transforms: global is mapped to local, so is easier to process.
Time/frequency trade-offCan we construct a function f whose energy is both well localised in time and whose FT has energy concentrated in a small frequency neighbourhood?Note that, in general, to try to reduce the time spread of f, we might scale time by some s<1 while keeping its energy constant:
)(ˆ)(ˆ that Note
that so ,)(1)( 22
sufsuf
ffstf
stf
s
ss
=
==
Since the FT of the scaled function is inevitably dilated by the scale parameter, we lose in frequency what we gain in time. In short, scaling doesn’t address the problem.
Is there a way to beat this problem? If not, what is the best trade-off we can manage?
Gabor’s great idea
• The windowed FT• Gabor “atoms”
– sinusoids multiplied by a Gaussian “window”• Uncertainty principle & Heisenberg map• Owens-Morrone-Kovesi theory of feature
detection– Analytic signal, Hilbert transform and
quadrature filters
The windowed FT
The windowed FT:
The function g is called a “window” function – it localises the Fourier integral in the neighbourhood of t = t0.
Various windowing functions have been proposed; but the most common is the Gaussian, as used by Denis Gabor(1950)
∫∞
∞−
−= dtettgtfutf tju0)()(),( 000S
Examples of the Windowed FT
]2
,2
[ intervalfrequency over the spread isenergy Its
)(ˆ),(
:(WFT) FT windowedhasIt
).(2)( FT has )( wavesinusoidalA
00
0)(
00
0
00
0
0
uu
uujt
tjuu
uu
uugeutf
uuufetf
σσ
πδ
+−
−=
−==
−S
Sinusoidal wave
Dirac function
]2
,2
[ intervalfrequency over the spread isenergy Its
)(),(
:(WFT) FT windowedhas )()(function DiracA
11
100
1
01
1
tt
ujt
t
tt
ttgeutf
tttf
σσ
δ
+−
−=
−=
S
Gabor functions
( )Gaussian. a toaccording off dies that sinusoid a is This
)(
:frequency at the modulated is )( window(Gaussian) symmetric and real a
0,
0
0
0
00ttGetg
uttg
tjuut −=
−
tCetg t 5cos)( 2/2−=
Gabor functions are sometimes called Morlet wavelets, after the French mathematician who independently discovered them
Gabor functions have uniform resolution in the
Heisenberg plane
dttgt
dttgtt
utg
utt
∫∫
=
−=22
2
,2
02
00
)(
)()(
, oft independen is of spread timeThe
00σ
uniformly.it they tile:planeg Heisenber theacross resolutionconstant a have functionsGabor
).,( oft independen is that ),(sizea has functionGabor a ofbox g Heisenber that thefollowsIt
.ˆ FTfor the holdsproperty similar A
00ut
, 00
ut
g ut
σσ
Gabor Transform in Heisenberg space
t
ω
σξ
σtThe time and frequency samples are windowed by a Gaussian whose standard deviations are
The Heisenberg box of a Gaborfunction has a size (σt,,σξ) that is independent of the space time point (t0,ξ0)
They have constant resolution across scale space, so limited usefulness
Uncertainty principle
∫
∫
∫
∫
−=
−=
=
=
duufuuf
f
dttfttf
f
duufuf
uuf
dttftf
ttf
u
t
22
22
222
2
2
2
22
)(ˆ)(2
1ˆ of Variance
)()(1 of Variance
)(ˆ2
1 over ˆ of Average
)(1over of Average
πσ
σ
π
Heisenberg uncertainty:4122 ≥ut σσ
The inequality is an equality if and only if
complex are , where,)(2)( baeaetf utbjut −−=
These modulated Gaussians are called Gabor chirps
Linear chirp
( )
phase. theof derivative theas defined isfrequency ousinstantane The
.2)(frequency at maximum its reacheshat Gaussian t a is ),(, timefixed aFor 41
4
),( ),(
is chirplinear theof spectogram the
1)( indowGaussian w aFor
ith time.linearly w increases that frequency" ousinstantane"an has )( chirplinear The
00000
41)2(2
1
42
2
20000
2
41
2
42
200
2
2
2
2
attuutPt
ea
utfutP
etg
etf
f
aatu
f
t
jat
=
⎟⎟⎠
⎞⎜⎜⎝
⎛+
=
=
=
=
+−
−
−
σσ
σ
σπσ
πσ
S
time
Instantaneous frequency
The thickness of the line is intended to indicate Gaussian window
Spectogram of a more complex signal
Top: the signal
Spectogram of the signal, which can be seen to contain a linear chirp, a quadratic chirp whose frequency decreases, and two modulated Gaussian functions (the black spots)
The lower figure shows the complex phase of the WFT of the signal in regions where the modulus of the spectogram is non-zero.
t
Earlier, we looked at step edge detection
However, even a moderately complex real world image has a wide variety of intensity change types
Feature detection
• There’s more to images than steps and corners– ramps, thin lines, and complex composite changes, ...
• The performance of feature detectors based on idealised models of a step/ridge degrade poorly on other kinds of features to which the Human Visual System is evidently sensitive
• Feature detectors based on amplitude of the gradient (local energy) are not invariant to contrast or brightness
Robyn Owens’ analysis of the Fourier components of “features”
Step up: all Fourier components have phase
zero
Ridge up and down: all Fourier components have phase π/2 or 3π/2
a feature is defined as a point where there is local phase congruency
Phase congruencyLocally, a one-dimensional signal can be expressed as:
PC is amplitude-weighted, mean local
phase
phase. eous)(instantan local theis )(and signal, theof (energy) amplitude eous)(instantan local theis )( where
)(cos)()(
ttA
ttAtI
n
n
nnn
φ
φ∑=
Note that, as we saw earlier, the concept of local phase depends on “windowing” the signal. It also requires that we be able to estimate the local phase at each time point t. We discuss that on the next slide
∑∑ −
=∈
nn
nnn
A
tAtPC
ttf))(cos(
max)(by defined is
point timeat the )( signal theof congruency phase The
]2,0[
φφ
πφ
Phase “congruency”
11,φA
22 ,φA
nnA φ,∑
iiiA φcos
∑i
iA
The ratio of the amplitude-weighted sum of phasors (the red arrow) to the sum of amplitudes (the green arrow) is a measure of “congruency”
(a) original image
(b) PC feature map
(c) Canny edge strength image
(d) raw PC image
Boat image. Note the faithful reconstruction of the rigging
(a) original image
(b) PC feature map
(c) Canny edge strength image
(d) raw PC image
Estimating local phase• Suppose that we have “windowed” the signal at
time point t for example by localling blurring with a Gaussian
• To estimate phase, we need to calculate something like sin, cos locally – this suggests using even and odd Gabor functions
• However, Gabor filtered signals have negative frequencies and non-zero DC – these cause problems in practice
• Generally, we seek a quadrature pair of filters –pairs of filters whose Fourier transforms are rotated by π/2
Estimating phase
• Define the Hilbert Transform• Hilbert transform corresponds to rotation
by π/2 in the Fourier domain – Just like sine, cos – which is why the Hilbert
Transform is key to estimating local phase• Define the Analytic Signal• Finally, show how to estimate (local)
phase
Hilbert transform)( ansformFourier tr uf
)(ˆ transform
Hilbert of ansformFourier tr
ufH)sgn()()(ˆ :have We
)sgn( function step
Heaviside theis 1 of ransform Fourier t theSince
1)(1
1)(1
)(1)(
:by defined is)( functiona of )( * The
uufuf
ut
ttf
dt
f
dt
ftf
tftfTransform Hilbert
H
H
H
⋅=
⎟⎠
⎞⎜⎝
⎛⎟⎠⎞
⎜⎝⎛∗=
−⋅=
−=
∫
∫∞
∞−
∞
∞−
π
ττ
τπ
τττ
π
Hilbert transform rotates by π/2 in the Fourier domain
( )( ) irH
riH
ir
ff
ff
ufjufuf
ˆˆ
ˆˆ
then),(ˆ)(ˆ)(ˆLet
−=
=
+=
x
x’
yy’
θ
⎥⎦
⎤⎢⎣
⎡⎥⎦
⎤⎢⎣
⎡−
=⎥⎦
⎤⎢⎣
⎡yx
yx
θθθθ
cossinsincos
''
The case of interest is when θ=-π/2
Recall rotation in the plane:
Analytical Signal & Hilbert Transform
)()()(tan phase local
)()()(energy local
define
tous enables This .2
by rotated isbut )( as magnitude same thehas
)(ˆ ransform Hilbert t thedomain, Fourier thein that recall We
)( of Transform Hilbert theis )( where)()()(
:)( theusing defined are)( signala of phase) ,(amplitude properties local theally,Mathematic
22
tftft
tftftA
uf
uf
tftftjftftf
tftf
H
H
H
H
HA
A
=
+=
−=
φ
π
SignalAnalytic
Analytic signal
)()(ˆ)(ˆ Evidently, usignufufH ⋅=
Original signal FT of f
Hilbert transform of f
Analytic signal for f
Analytic signal and sine/cosineThe Fourier representation of a signal f(x) can be defined as :
∫∞
+=0
)(cos()()( ωωφωω dxAxf
Now include the imaginary part:
∫∫
∫∞∞
∞+
+++=
=
00
0
))((
))(sin()())(cos()(
)()(ˆ
ωωφωωωωφωω
ωω ωφω
dxAjdxA
deAxf xj
Evidently, the imaginary part is a phase shifted version of the real part. Again, the imaginary part is the Hilbert Transform of f(x)
Implementation of Phase Congruency (Kovesi)
Need to estimate/correct for noise and estimate frequency spread
)01.0 (typically ,)(
)()( Calculate 3.
)()()(
)()(
)()( :Calculate 2.
)()()(
)()()(),()()(
,...,1for 1.function.Gabor (log) odd andeven the)(),(by Denote
22
22
max
=+
=
+=
=
=
+=
∗=∗=
=
∑
∑
∑
εε
nn
nn
nn
nnn
onn
enn
thon
en
tAtWtPC
tHtFtW
tOtH
tEtF
tOtEtA
tMtftOtMtftE
nnntMtM
Calculation of phase congruency via the FFT
(a) original image
(b) PC feature map
(c) Canny edge strength image
(d) raw PC image
(a) original image
(b) PC feature map
(c) Canny edge strength image
(d) raw PC image
(a) original image
(b) PC feature map
(c) Canny edge strength image
(d) raw PC image
Demonstrating invariance to contrast. Canny disappears in shadow regions
(a) original image
(b) PC feature map
(c) Canny edge strength image
(d) raw PC image
PC: from signals to images
So far, we have described how to compute the phase congruency PC(x) at each point x of a signal. Now we ask how to extend the idea to images.
The immediate problem is that the Hilbert transform is only defined for signals, not for images.
Phase congruency of imagesAt each image location (x,y), we can choose a direction, θ, and interpolate the signal S(θ;x,y) in direction θpassing through x,y. We can then compute the PC(θ;x,y)
Finally, we can define the phase congruency for the image point (x,y) from some quantity computed from
{PC(θi;x,y):i=1..n}
Kovesi suggested using the averageof the directional phase congruences. That is what is shown in the results presented thus far.
Aerials are not steps, and are hard to detect
Output from a PC implementation developed by Brady and Liang in which directional phase congruency responses are combined in a rule-based manner
Phase congruency• Good news:
– responds to a set of diverse features– geometrically and photometrically invariant
• Bad news: – Being a normalised quantity, PC responds strongly to noise– PC is only meaningful if there is a spread of frequencies in a
signal; but it is not obvious how to define this spread– PC seems to require a different interpretation of scale,
suggesting that high-pass filtering rather than low-pass filtering should be used.
– PC can be defined in 1D, but, until recently it was not obvious how to calculate it in 2D, 3D, ...
Noise sensitivity of phase congruency
The Windowed FT seems to solve all our problems of time/space trade-off
…. or does it?
Sine & discontinuity
The Fourier Transform OR THE WINDOWED FOURIER TRANSFORM consists of two delta functions, indicating the single dominant signal. The wiggles show that it detects thatthere is a discontinuity; but cannot say where
Wavelets find the Sine & discontinuity
scal
e
timeThe wavelet transform indicates thatthere is a discontinuity and shows where
Wavelets enable the discontinuity to be localised accurately
Note that the discontinuity appears at all scales, with higher SNR at coarse scales; better localisation at finer scales. Evidently, the feature exhibits congruency
This is a Scalogram. The vertical axis shows the scale nof analysis. The horizontal axis is time t. Dark means low values, light means high.
scal
e
time
Example wavelets
db10
A wavelet is a function of effectively limited duration whose average is zero
Wavelet analysis is breaking up a given signal into shifted and scaled versions of a basic “mother” wavelet
db2 Biorthogonal2.6
Meyer
Wavelet definition
A wavelet is a function with zero average
which is dilated with a “scale parameter” s, and translated by some t0.
The function shape ω(t) is called the mother wavelet
∫∞
∞−
= 0)( dttψ
)(1)( 0,0 s
tts
tst−
= ψψ
Wavelet transformThe wavelet transform of some signal f(t) at scale s, and at position t0 is defined from the correlation:
dtstttf
sstf ∫
∞
∞−
⎟⎠⎞
⎜⎝⎛ −
= 0*0 )(1),( ψW
filters. bandpass dilated with ransform wavelet t thecomputesn convolutio Thefilter. bandpass a offunction transfer theis ˆ that appearsit
,0)(ˆ Since ).(ˆ)(ˆ is of FT The
1)(
where,),( that Notice
*
*
0
ψ
ψψψψψ
ψψ
ψ
∫∞
∞−
===
⎟⎠⎞
⎜⎝⎛ −=
∗=
dttsusu
st
st
fstf
ss
s
sW
Continuous Wavelet Transform
∫∞
∞−
= dttpositionscaletfpositionscaleCWT ),,()(),( ψ
Wavelet scaling
1)()(
==
attf ψ
21
)2()(
=
=
a
ttf ψ
41
)4()(
=
=
a
ttf ψ
Five easy steps to a CWT1. Choose a wavelet and compare it to a section at the start of a signal
2. Calculate a number C that represents how closely correlated the wavelet is with this section of the signal. More precisely, if the signal energy and wavelet energy are both equal to 1, C may be interpreted as a correlation coefficient. Note that this depends on the wavelet.
Five easy steps to a CWT3. Shift the wavelet to the right and repeat steps 1, 2 until you have covered the whole signal.
4. Scale (stretch) the wavelet and repeat 1-3.
5. Repeat steps 1-4 for all desired scales
CWT scalogram
CWT scalogram as a surface
CWT of the noisy signal shown at the top, scalogram in the second row, final approximation signal in third row, and loci of maxima of scalogram over scale at bottom.
This uses the db4 wavelet
CWT of the noisy signal shown at the top, scalogramin the second row, final approximation signal in third row, and loci of maxima of scalogram over scale at bottom.This uses the Haar wavelet
Scale and frequency
• Low scale a compressed wavelet rapidly changing details high frequency ω
• high scale a stretched wavelet slowly changing, coarse features low frequency ω
Discrete wavelet transform
• Problem is that the CWT is a fair amount of work – it is impractical in most circumstances
• It turns out – remarkably – that if we limit ourselves to positions & scales based on powers of two – so called dyadic scales – we obtain the discrete wavelet transform (DWT)
• There is a fast DWT equivalent to the FFT• The first Fast DWT was developed by Mallat
(1987). Several have been developed since.
Approximation & detail
• approximations are high scale (low frequency) components of a signal
• details are the low scale (high frequency) components
• do this successively at several scales, as defined by DWT
Downsampling
Using complementary filters doubles the amount of data, so over a range of scales you drown in data.
Downsampling by 2 maintains the same amount of data at each scale. With a proper choice of filter pair, this yields the DWT.
We remember that this is the key to the FFT!
Downsampling II
Discrete multiresolution approximations aj[n] at scales 2j computed with cubic splines
Approximation Components
Detail Components
Original Signal
Discrete Wavelet Transform
We’ll see later that this can be efficiently extended to 2, 3, …dimensions
DWT of the signal shown in first row. The approximation at level 5 is shown in the second row, and the five detail signals are shown in rows 3 to 7.
Here we use the Haar wavelet.
DWT of the signal shown in first row. The approximation at level 5 is shown in the second row, and the five detail signals are shown in rows 3 to 7.
Here we use the db4 wavelet.
Extension of the FWT to 2 dimensions
The extension to 3D, 4D is straightforward.
Mallat’s depiction of the 2D FWT
Decomposition & reconstruction
This is the basis for efficient wavelet coding and decoding of signals, used increasingly in JPEG2000 and MPEG2000
Coding fidelity
Flushing the first several D’s can reduce the coding length, without sacrificing significant information
Discrete Wavelet Transform
We’ll see later that this can be efficiently extended to 2, 3, …dimensions
Wavelet bases & the fast WT1. Function spaces, orthonormal bases of functions, linear
approximation & projection 2. An approach is developed to multiresolution
approximations of a signal. This uses a sequence of subspaces of .
3. We show how approximation can be summarised by a single function θ.
4. This leads to the development of a scaling function ø(t) from which everything else is derived!
5. ø(t) yields a low-pass filter h(t), from which a high-pass filter g(t) is developed. Mallat calls the pair g,h a conjugate mirror pair.
6. finally the wavelet ψ(t) is derived from ø,g,h and the scheme leads directly to the Fast Wavelet Transform.
{ }Zjj ∈
V )(2 ℜL
Example of g, h
Solid line is the FT of h on [-π, π] and the dotted line is the FT of g. Note how h is approximately low pass and g is high pass. These are the filters from a cubic spline multiresolution wavelet.
Function spacesThe familiar idea of a vector space can be generalised to functions.For present purposes it is possible to skip most of the mathematics(of Banach, Hilbert, and Riesz spaces).
We need to define an inner product of two functions f(t), g(t). This is usually taken to be: ∫
∞
∞−
>=< dttgtfgf )()(, *
with the resulting norm: 21
, >=< fff
00, all ,0,
,,,,,
2121
=⇔=
>≥<><+><>=+<
>>=<<
ffgfgf
gfgfgfffggf
µλµλ
Orthonormal function bases{ }
1 ,0,
if lorthonorma is )( functions ofset A
=
≠>=<
= ∈
n
nm
Znn
nmt
φφφφB
nn
nnff
αφα s"coordinate"for :as expressed becan class in the function every if functions of class afor basis a is
∑=B
∑ ><=n
nnff φφ, Evidently,
{ } Znjnue ∈ is basis lorthonormaFourier The
Linear approximation { }
∑−
=
∈
><=
=
1
0
,
:bygiven is basis theof functions first over the function a ofion approximatlinear A
basis.function lorthonormaan is Suppose
M
nnnM
Znn
ff
Mf
φφ
φB
Since the basis is orthonormal, the approximation error is given by the sum of the remaining squared inner products:
∑∞
=
><=−=Mn
nM fffM 22 ,)( φε
the Fourier basis yields efficient linear approximations of uniformly smooth signals, with a projection over the M lower frequency sinusoids.in a wavelet basis, the signal is projected over the M larger scale wavelets, equivalent to approximating the signal at a fixed resolution.
Multiresolution approximationSuppose that the wavelet basis that we seek is:
{ } 22
21)( thatrecall weand )( ,),(, 2 ⎟⎟
⎠
⎞⎜⎜⎝
⎛=
∈ j
j
jnjZnjnjnt-tt ψψψ
Then, for any signal f(t) that we wish to analyse using the waveletbasis, we are interested in the partial sums of wavelet coefficients:
∑∞
−∞=
><n
njnjf ,,, ψψ
The jth term of this sum can be interpreted as the difference between the approximations of f(t) at the resolutions 2-(j+1) and 2-j.
Multiresolution approximations compute the approximation of thesignal f(t) as orthogonal projections on different spaces Vj. The multiresolution approximation is composed of embedded grids ofapproximation.
Approximation spaces
jV1−jV
1+jV
{ }
1 scalecoarser aat ionapproximat thecompute n toinformationecessary
theall contains scale aat tion (approxima
)2()(2
)(
, ).( of
subspaces of sequence a develop We
1
12
+
∈−⇔∈
∈⎟⎠⎞
⎜⎝⎛⇔∈
⊆∀ℜ
+
+
∈
j
j
ntktk
tktk
jL
jj
j
jj
jj
Zjj
VV
VV
VV
V
{ }0 subspace resolutionfinest for the basis a are
n)-(t ons translatiits that so )(function a is ThereV
Znt ∈θθ
Examples of approximation spaces
• Piecewise constant approximations: the space Vj is the set of functions f(t) that are constant in the interval [n2j,(n+1)2j]. The approximation at resolution 2j is the closest piecewise constant function on intervals of size 2j . In this case θ(t) is the constant box function.
• Shannon approximations: the space Vj is the set of frequency band limited functions whose FT has a support included in [-2-jπ , 2-jπ ]. It turns out that the spaces can be defined by sinc functions of various frequencies, indeed, θ(t)= (sin πt)/ πt.
• Spline approximations: the provide smooth approximations with fast decay. The maths is a little more complex.
Scaling function øSuppose that we have been given the function θ(t). We use it, rather we use its FT, to define the scaling function ø(t) whose dilations and translations form a basis for the sequence of approximation spaces Vj.
{ } jZnnj
jjnj
k
ntt
ku
uu
V of basis lorthonormaan is shown that becan It 22
1)(
)2(ˆ
)(ˆ)(ˆ Define
,
,
2
∈
∞
−∞=
⎟⎠⎞
⎜⎝⎛ −
=
+
=
∑
φ
φφ
πθ
θφ
For piecewise constant and Shannon approximations, ø=θ. However, this is rare …
Approximation using øWith this basis, the approximation of the signal f(t) over Vj is given by:
∑∞
−∞=
><=n
njnjffPj ,,,)( φφV
Denote the inner products by aj[n] since they provide an approximation of f at the scale 2j Given the scaling function ø, the approximation can be rewritten as a convolution:
)2(
22
21)(][
, nf
dtnttfna
jnj
j
j
jj
φ
φ
∗=
⎟⎟⎠
⎞⎜⎜⎝
⎛ −= ∫
∞
∞−
j
j
tf
nau
2 intervalsat sampled ,)( of version filtered pass-low a is
][ion approximat that theso ,],[in edconcentrat typicallyis )(ˆ FT theofenergy The ππφ −
Low pass filter h
{ }
)(ˆ)(ˆ2
1)2(ˆ :domainFourier In the
)(,22
1][ where
)(][22
1
:for basis a is )( Since22
1 that and )(hat Remember t
0
010
uuhu
nttnh
ntnhtnt
tt
n
Zn
φφ
φφ
φφ
φ
φφ
=
>−⎟⎠⎞
⎜⎝⎛=<
−=⎟⎠⎞
⎜⎝⎛
−
⊆∈⎟⎠⎞
⎜⎝⎛∈
∑∞
−∞=
∈ V
VVV
This relates a dilation of ø by 2 to its integer translations. The sequence h[n] can be regarded as a discrete filter.
Examples of hPiecewise constant approximations (eg. Haar)
⎪⎩
⎪⎨⎧ ==
>−⎟⎠⎞
⎜⎝⎛=<=
otherwise0
1,02
1][
thatfollowsit ,)(,22
1][ sinceThen .]1,0[
nnh
nttnh φφφ 1
Polynomial cubic splines
2cos
cos212
cos212
)(ˆ)2(ˆ2)(ˆ
2
21
2
2
uu
u
uuuh
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
+
+=
=φφ
Orthogonal complements
0, and
:satisfy , on to sprojection its ,function any for Then,
: of complement orthogonal
thedefine now We., scaleeach at that know We
1
1
1
>=<+=
∈
⊕=
⊆
−
−
−
fPfPfPfPf
f
j
jjjj
jjj
jjjjj
jj
WVWV
WVV
WVVVW
VV
Wavelet bases & the fast WT1. Function spaces, orthonormal bases of functions, linear
approximation & projection 2. An approach is developed to multiresolution
approximations of a signal. This uses a sequence of subspaces of .
3. We show how approximation can be summarised by a single function θ.
4. This leads to the development of a scaling function ø(t) from which everything else is derived!
5. ø(t) yields a low-pass filter h(t), from which a high-pass filter g(t) is developed. Mallat calls the pair g,h a conjugate mirror pair.
6. finally the wavelet ψ(t) is derived from ø,g,h and the scheme leads directly to the Fast Wavelet Transform.
{ }Zjj ∈
V )(2 ℜL
Low pass filter h
{ }
)(ˆ)(ˆ2
1)2(ˆ :domainFourier In the
)(,22
1][ where
)(][22
1
:for basis a is )( Since22
1 that and )(hat Remember t
0
010
uuhu
nttnh
ntnhtnt
tt
n
Zn
φφ
φφ
φφ
φ
φφ
=
>−⎟⎠⎞
⎜⎝⎛=<
−=⎟⎠⎞
⎜⎝⎛
−
⊆∈⎟⎠⎞
⎜⎝⎛∈
∑∞
−∞=
∈ V
VVV
This relates a dilation of ø by 2 to its integer translations. The sequence h[n] can be regarded as a discrete filter.
High pass filter g
passlow is since pass, high is Evidently,]1[)1(][
:show thattorward"straightfo" isIt 1
hgnhng n −−= −
The high pass filter g is constructed directly from the low pass filter hfrom its FT.
)(*ˆ)(ˆ :domain Fourier thein )( define We π+= − uheugtg ju
Example of g, h
Solid line is the FT of h on [-π, π] and the dotted line is the FT of g. Note how h is approximately low pass and g is high pass. These are the filters from a cubic spline multiresolution wavelet.
wavelet ψ
{ } jZnnj
j
j
jnjntt
uugu
t
W of basis lorthonorma an is that proved be canIt 22
21)(
2ˆ
2ˆ
21)(ˆ
:by )( function thedefine ely)(suggestivNext
,
,
∈
⎟⎟⎠
⎞⎜⎜⎝
⎛ −=
⎟⎠⎞
⎜⎝⎛
⎟⎠⎞
⎜⎝⎛=
ψ
ψψ
φψ
ψ
High pass filter g and wavelet ψ
>−⎟⎠⎞
⎜⎝⎛=<
−=⎟⎠⎞
⎜⎝⎛ ∑
∞
−∞=
)(,22
1][
)(][22
1
][ :tscoefficien thedefinecanwe together,all thisGathering
nttng
ntngtng
n
φψ
φψ
Schematic of the fast wavelet transform (Mallat)
g is the high pass filter constructed from the mother wavelet ω and the scaling function υ. h is the corresponding low-pass filter
Representation of the Wavelet Transform
t
s
Ψ/s
sΨNote how the sampling in time scales with frequency: as frequency decreases, temporal sampling increases, and conversely
Shannon Fourier
Gabor / STFT Wavelet
Heisenberg “boxes”
Zoom Zoom Zoom
Debauchies’ wavelets: compact support and orthonormal
Debauchies’ wavelets are approximately fractal, hence their use in texture analysis
Segmenting an aerial image on the basis of texture: db4
Xie and Brady, 1994
Coifman’s wavelets
Almost symmetric, good frequency and space localisation, compact support, orthogonal. Excellent for picking out sharp, spatially localised features
Anisotropic Diffusion
Wavelet Analysis
Gaussian Scale-Space
Calcifications & vessel are much better preserved
Recall Phase Congruency• We used the Hilbert Transform to define local
phase• From that we defined phase congruency at each
point f(t) of a signal• We extended the definition to phase congruency
at each point I(x,y) of an image, by computing the directional phase congruency in a set of directions θi and then combining the results
The problems are (a) that this is intrinsically inefficient; and (b) is prone to interpolation errors in forming the directional signals.
Can we compute Phase Congruency of a 2- or 3-dimensional image without having to resort to one dimensional signal-like slices?
Monogenic signalPhase congruency
[ ]
[ ])(),( from phase
Localbandpass. is )( where)(*)()(
signal thesmooth practice, Inamplitude phase, local Define
)(),( signal Analytic)( ansformHilbert tr
tftf
tbtbtftf
tftftf
Hbb
b
H
H
=
( ) ( )[ ]
image. the in norientatio local phase local
(energy), amplitude local compute we),(),,(),,(
from then),,(is image smoothed bandpass theIf
).,(),,(domain space theinly equivalent
),,(),,( kernels, nconvolutiomoreor twouses transformRiesz
21
21
21
and
yxIhyxIhyxIyxI
yxhyxh
vuHvuH
bbb
b
∗∗
Monogenic signal
( )
( )
sinea is
and domain, Fourier thein cosinea is that Note
2),(ly equivalent,),(
2),(ly equivalent ,),( :where
),(),(
),(
22
22
23
222222
23
221221
2
1
vuv
vuu
yx
yyxhvu
vvuH
yx
xyxhvu
uvuH
vuHvuH
vuM
+
+
+=
+=
+=
+=
⎥⎦
⎤⎢⎣
⎡=
π
π
Computing the monogenic signal),( Image yxI
),(),(),(filter bandpass
yxbyxIyxI b ∗=
),(),(1 yxIyxh b∗ ),(),(2 yxIyxh b∗
phase local )()(),(tan
2
1
b
b
IhIhyx
∗∗
=θ
),(),(),( yxbyxIyxI b ∗=
energyor amplitude local )()(),( 2
22
12
bbb IhIhIyxE ∗+∗+=
image thein norientatio local )()(
),(tan2
22
1 bb IhIhIyx
∗+∗=φ
Monogenic triple of filters
Bandpass filter is a difference of Gaussians
h1*B h2*B
Though this generally works well, the difference of Gaussians is not the only bandpass filter that can be used …
Mammogram segmentation
50 100 150 200 250 300 350
50
100
150
200
250
300
350
400
450
500
Original mammogram image (log scale)
50 100 150 200 250 300 350
50
100
150
200
250
300
350
400
450
500
0.5
1
1.5
2
2.5
3
Local phase from log Gabor filters and monogenic signal
50 100 150 200 250 300 350
50
100
150
200
250
300
350
400
450
500
orientation
50 100 150 200 250 300 350
50
100
150
200
250
300
350
400
450
500 -3
-2
-1
0
1
2
3
50 100 150 200 250 300 350
50
100
150
200
250
300
350
400
450
500
Local energy, with breast edge marked
50 100 150 200 250 300 350
50
100
150
200
250
300
350
400
450
500
Phase < 90 degrees
50 100 150 200 250 300 350
50
100
150
200
250
300
350
400
450
500
Phase > 90 degrees
Motion at points in the heart
Displacement errors for two successive ultrasound images of the heart, prior to registration (left) and post registration (right). Near “features”, the error is less than a pixel.
Affine invariance
An image with the interesting points added: blue = high symmetry; red = high antisymmetry
Affine transformed house image with interest points superimposed (circles) and matched (stars).
Local symmetry
Points of local symmetry Locally “interesting” points
Blue = high symmetry; red = high antisymmetry