Orthogonal Transforms

Fourier

Review• Introduce the concepts of base functions:

– For Reed-Muller, FPRM– For Walsh

• Linearly independent matrix• Non-Singular matrix• Examples• Butterflies, Kronecker Products, Matrices• Using matrices to calculate the vector of spectral

coefficients from the data vector

Our goal is to discuss the best approximation of a function using orthogonal functions

Orthogonal Orthogonal FunctionsFunctions

Orthogonal FunctionsOrthogonal Functions

Note that these are arbitrary functions, we do not assume sinusoids

Illustrate it for Walsh and RM

Mean Mean Square Square ErrorError

Mean Square ErrorMean Square Error

Important result

• We want to minimize this kinds of We want to minimize this kinds of errors.errors.

• Other error measures are also used.Other error measures are also used.

Unitary Unitary TransformsTransforms

Unitary TransformsUnitary Transforms• Unitary Transformation for 1-Dim. Sequence

– Series representation of

– Basis vectors :

– Energy conservation :

}10),({ Nnnu

1

0

10),(),()(N

n

NknunkakvAuv

) ( *1 matrixunitaryAAwhere T

1

0

** 10),(),()(N

n

NnkvnkanuA vu

TNnnka }10),,({ * *ka

22 |||||||| uvuv A

)|||||)(||)(||||| ( 21

0

2*1

0

22 uuuuuv T*T*

N

n

TN

k

nuAAkv

Here is the proof

• Unitary Transformation for 2-Dim. SequenceUnitary Transformation for 2-Dim. Sequence– Definition :

– Basis images :– Orthonormality and completeness properties

• Orthonormality :

• Completeness :

1,0),,(),(),(1

0

1

0,

NlknmanmulkvN

m

N

nlk

1,0),,(),(),(1

0

1

0

*,

NnmnmalkvnmuN

k

N

llk

)},({ *, nma lk

),(),(),( ''*

,

1

0

1

0, '' llkknmanma

lk

N

m

N

nlk

),(),(),( ''''*,

1

0

1

0, nnmmnmanma lk

N

k

N

llk

• Unitary Transformation for 2-Dim. SequenceUnitary Transformation for 2-Dim. Sequence– Separable Unitary Transforms

• separable transform reduces the number of multiplications and additions from to

– Energy conservation

)()(),(, nbmanma lklk

Tl

N

m

N

nk AUAVnanmumalkv

)(),()(),(1

0

1

0

***1

0

1

0

* )(),()(),( VAAUnalkvmanmu Tl

N

k

N

lk

)( 4NO )( 3NO

1

0

1

0

21

0

1

0

2 |),(||),(|N

k

N

l

N

m

N

n

lkvnmu

iondecorrelat and compactionenergy

83.0(1)σ(0)σ

(0,1)ρ

is and between n correlatio theand

in energy 91% then , 0.95ρ if2

ρ31(1)σ2ρ31(0)σbut

ddistributeequally isenergy total;(1)σ1(0)σ

then

where

d trasformeis r mean vecto zero 12A

then ,Consider

vvv

2v

2v

2u

2u

]Ε[v(0)v(1)

v(1)v(0)

v(0)

2ρ312

ρ2

ρ2

ρ31R

1ρ

ρ1Ru

31

13

2

1v

u

)(AARRuAv

v

u

t*uv

Properties of Unitary TransformProperties of Unitary Transformtransform

Covariance matrix

Example of arbitrary Example of arbitrary basis functions being basis functions being

rectangular wavesrectangular waves

This determining first function determines next functions

10

Small error with just 3 coefficients

This slide shows four base functions multiplied by their respective coefficients

End of example

This slide shows that using only four base functions the approximation is quite good

Orthogonality Orthogonality and and

separabilityseparability

Orthogonal and separable Image Orthogonal and separable Image TransformsTransforms

Extending Extending general transformsgeneral transforms to 2- to 2-dimensionsdimensions

separable

Forward transform

inverse transform

Fourier Fourier Transforms in Transforms in new notationsnew notations

1. We emphasize generality2. Matrices

Fourier TransformFourier Transform

separable

Extension of Fourier Transform to Extension of Fourier Transform to two dimensionstwo dimensions

1,N0,1,nwkvN

nu

1,N0,1,kwnuN

kv

ew

Nkwnuv(k)

,N-10,1,2,nn

N

k

knN

N

n

knN

jN

N

n

knN

N

,)(1

)(

,)(1

)(

ansformunitary tr be oproperly t scalednot

where

1,,1,0)(

is , of DFT The

1

0

1

0

1

0

2

)u(

Discrete Fourier Transform (DFT)Discrete Fourier Transform (DFT)

New notation

allymathematic handle easy to

11

)3(

)2(

)1(

)0(

1

1

1

1111

2

1

)3(

)2(

)1(

)0(

For

2

14

24

34

24

04

24

34

24

14

knjknN

NeN

wN

u

u

u

u

www

www

www

v

v

v

v

4N

F

uFv

additions 8 , tionsmultiplica 4:ionfactorizat

additions 12 , tionsmultiplica 16 :direct

)3(

)2(

)1(

)0(

010

001

010

001

100

100

001

001

)3(

)1(

)2(

)0(

2 where),log()(

algorithmFast .2

since,

.1

24

24

04

04

34

14

04

04

22

u

u

u

u

w

w

w

w

w

w

w

w

v

v

v

v

NNNN nOΟ

I)F(FFF)(F

FFt*1*t*

*1

Fast Algorithms for Fourier Fast Algorithms for Fourier TransformTransform

2

Task for students:

Draw the butterfly for these matrices, similarly as we have done it for Walsh and Reed-Muller Transforms

Pay attention to regularity of kernels and order of columns corresponding to factorized matrices

transformSine Cosine, Fourier,

.operations requires then

,say elements, nonzero few ajust

withmatrices are)(,,2,1, where

ionfactorizat: if general,In

rpN

N r

Npi

xAv

A

AAAA

i

p21

Fast Factorization Algorithms are Fast Factorization Algorithms are general and there is many of themgeneral and there is many of them

• 1-dim. DFT (cont.) – Calculation of DFT : Fast Fourier Transform Algorithm

(FFT)

• Decimation-in-time algorithm

NkNengeeng

Nkengeeng

eengeng

engeng

engkG

N

N

NN

N

N

N

N

N

N

N

N

N

NN

n

knj

N

kj

n

knj

n

knj

N

kj

n

knj

n

knj

N

kj

n

knj

n

N

nkj

n

N

nkj

N

n

N

knj

2 ,)12()2(

120 ,)12()2(

)12()2(

)12()2(

)()(

1

0

221

0

2

1

0

221

0

2

1

0

221

0

2

1

0

122

1

0

22

1

0

2

2

2

22

2

2

2

2

2

2

2

2

2

22

Derivation of decimation in time

Decimation in Decimation in Time versus Time versus

Decismation in Decismation in FrequencyFrequency

• 1-dim. DFT (cont.) – FFT (cont.)

• Decimation-in-time algorithm (cont.)Butterfly for Derivation of decimation in time

Please note recursion

• 1-dim. DFT (cont.) – FFT (cont.)

• Decimation-in-frequency algorithm (cont.)

12 ,)()(

2 ,)()(

1)()(

)()(

)()2(

)()(

21

0

2

2

1

0

2

2

1

0

2

2

1

0

22

2

2

1 21

0

2

1

0

2

2

2

2

2

2

2

2

2

kkeengng

kkengng

engng

eengng

engeng

engkG

N

nj

n

nkj

N

n

nkj

N

n

N

knjkN

n

N

knj

N

N

kj

N

N

n

N

knj

n

N

knj

N

n

N

knj

N

N

N

N

N

N

N

N

Derivation of Decimation-in-frequency algorithm

• 1-dim. DFT (cont.) – FFT (cont.)

• Decimation-in-frequency algorithm (cont.)

Decimation in frequency butterfly shows recursion

litydimensiona theincreasenot do

DFT thedefines completely ) v(and

12

,,1,)(Im,12

,,1,)(Re)0(

2N

Nkkv

Nkkvv

kkN

)v()(v*

– For a real sequencereal sequence, the DFT is conjugate symmetry

Conjugate Conjugate SymmetrySymmetry of of DFTDFT

Use of Fourier Use of Fourier Transforms Transforms

for fast for fast convolutionconvolution

1,,1,0,1w

unity. of rootsdistinct theof one iswthen

)exp(wLet

cc

cccc

ccc

matrixcircular aconsider usLet

*

*

2*

01

2N101N

1N10

Nk

N

j

kN

N

k

N

NN

C

Calculations for circular matrixCalculations for circular matrix

kN

N

k

N

kN

N

kN

NN

k

NNN

k

N

kN

NN

k

NN

Nk

NN

k

N

k

N

k

N

kN

NN

k

N

k

N

wcwcckw

wcwcckw

wcwcc

wcwcwckw

wcwcwcck

1*0

*21

1*

1*3

*12

2*

1*2

*01

*1

2*1

*0

*

1*1

2*2

*10

By multiplying

**

*

*

**

1*

2*

*

021

201

110

1*

2*

*

thus, orseigen vectdistinct are there

ofor eigen vectan is, and eeigen valuan is )(

)(

11

k

form,matrix aIn

Cww

w

Cw

xxwCw

kN

kk

Akkk

w

w

w

ccc

ccc

ccc

w

w

w

kN

N

k

N

k

NNN

N

kN

N

k

N

k

N

In matrix form next slideW * = Cw*

matrix diagonal;

1000

0

0010

000

)Fmatrix( ansformFourier tr inverse;

1

1

1

1

1

111111

where 1

42

21

*

N

ww

ww

Λ

w

DFTby eddiagonaliz is Cmatrix circular

or *1*

1**

Cww

AFFwwC 1

w * = Cw*

81

43,2,1,0

54,3,2,1,0 : assume

:nconvolutiolinear 1

0

LNM

Lhhhh

Nxxxxx

jnhjxnyn

j

0211202

01101

000

hxhxhxy

hxhxy

hxy

Here is the formula for linear convolution, we already discussed for 1D and 2D data, images

xHy

;

15

4

3

2

1

0

5830000

23000

12300

01230

00123

00012

00001

00000

187

6

5

4

3

2

1

0

x

x

x

x

x

h

hh

hhh

hhhh

hhhh

hhh

hh

h

y

y

y

y

y

y

y

y

Linear convolution can be presented in matrix form as follows:

xy c

1

0

H;

3

2

1

0

0123

3012

2301

1230

3

2

1

0

,4For

mod where

1,1,0,

nconvolutioCircular

x

x

x

x

hhhh

hhhh

hhhh

hhhh

y

y

y

y

N

Nnhnh

Nnjnhjxny

N

N

N

j

021

201

110

generalIn

hNhNh

hhh

hNhh

cH

As we see, circular convolution can be also represented in matrix form

ihH

kH

kjjjh

N-Nkjjh

kjjhkH

jhjh

wcwcwcckλ

c

N

N

j

N

jN

N

N

j

kN

NN

k

N

k

N

of DFT theis of eeigen valu

exp

1 to0 toequal is 1 to0 from sum;exp

exp

sequence periodic a is since , is H of rowfirst theThus

remember

21

0

1

0

2

21

0

c

1*1

2*2

*10

Important result

DFTfor algorithmfast

of DFT inverse;3

2

of DFT forward;1

100

00

10

000

where

and

now,

kXkH

kXkH

NH

H

H

xΛFF

;xFΛ

xxF

Λ

xΛFFy

HΛFFxHy

1

1

c1

c

Inverse DFT of convolution

padding zero;1,,,0

1,,1,0,~

padding zero;1,,,0

1,,1,0,~

then

22let

2,,1,0,

1,1,0,

1,,1,0,

nconvolutiolinear

MLi

Liihih

MNi

Niixix

LNM

LNiihixiy

Liih

Niix

m

filteringnconvolutiolinear for algorithmfast

of components 2 the toequal

exactly are ~ of components 2first the

~~ performthen

y

y

xHy c

LN

LN

•Thus we derived a fast algorithm for linear convolution which we illustrated earlier and discussed its importance.

•This result is very fundamental since it allows to use DFT with inverse DFT to do all kinds of image processing based on convolution, such as edge detection, thinning, filtering, etc.

2-D DFT2-D DFT

NNNN

and

NnmwwlkvN

nmu

NlkwwnmuN

lkv

N

k

N

l

lnN

kmN

N

m

N

n

lnN

kmN

2

*

1

0

1

0

1

0

1

0

log2 DFT D-1 2separable.2

where

notation spacevector

since.1

1,0,),(1

),(

1,0,),(1

),(

is DFTunitary D2

O

FF

vuuv

VFFU

FFFUFV**

t

F

FF

2-D DFT2-D DFT

1,0,,

~,~

,~

,~

,~

1,,0

1,0,,,~

1,,0

1,0,,,

~

1,,,,

n theoremconvolutiocircular D2 .4

1,0,,,

real is if symmetry, e3.conjugat

1

*

MnmlkYDFTnmy

lkUlkHlkY

MnmN

Lnmnmunmu

MnmN

Nnmnmhnmh

LNMnmunmhnmy

NlklNkNvlkv

u(m,n)

NNLLMM

N+L-1

M

MN+L-1

Circular Circular convolution convolution works for 2D works for 2D imagesimages

• 2-Dim. DFT (cont.)– example

image Lena 512512 a of DFT dim2

(a) Original Image (b) Magnitude (c) Phase

Circular convolution works for 2D imagesCircular convolution works for 2D images

So we can do all kinds of edge-detection, filtering etc So we can do all kinds of edge-detection, filtering etc very efficientlyvery efficiently

• 2-Dim. DFT (cont.) – Properties of 2D DFT

• SeparabilitySeparability

1,,0, ,),(11

),(1

0

1

0

NlkWnmfN

WN

lkFN

m

N

n

lnN

kmN

1,,0 ,),(11

),(1

0

1

0

Nm,nWlkFN

WN

nmfN

k

N

l

lnN

kmN

• 2-Dim. DFT (cont.) – Properties of 2D DFT (cont.)

• RotationRotation

sin ,cos ,sin ,cos lkrnrm

),(),( 00 Frf

(a) a sample image (b) its spectrum (c) rotated image (d) resulting spectrum

• 2-Dim. DFT (cont.) – Properties of 2D DFT

• Circular convolution and DFTCircular convolution and DFT

• CorrelationCorrelation

p q

C qnpmgqpfnmgnmf ),(),(),(),(

),(*),(),(),(

),(),(),(*),(

lkGlkFnmgnmf

lkGlkFnmgnmf

p q

Cfg qnpmgqpfnmgnmfnmR ),(),(),(),(),( *

),(),(),(),(

),(),(),(),(*

*

lkGlkFnmgnmf

lkGlkFnmgnmf

• 2-Dim. DFT (cont.) – Calculation of 2-dim. DFTCalculation of 2-dim. DFT

• Direct calculation

– Complex multiplications & additions :

• Using separability

– Complex multiplications & additions :

• Using 1-dim FFT – Complex multiplications & additions : ???

10

,10 ,),(),(

2

11

0

1

0

221 2

Nl

NkeenmflkF

N

m

N

n

lnN

jkmN

j

)( 22

21 NN

10

,10 ,),(),(

2

11

0

21

0

21 2

Nl

NkeenmflkF

N

m

kmN

jN

n

lnN

j

))(( 2121 NNNN

Three ways of calculating 2-D DFT

Questions to StudentsQuestions to Students1. You do not have to remember derivations but you have to understand the

main concepts.

2. Much software for all discussed transforms and their uses is available on internet and also in Matlab, OpenCV, and similar packages.

1. How to create an algorithm for edge detection based on FFT?

2. How to create a thinning algorithm based on DCT?

3. How to use DST for convolution – show example.

4. Low pass filter based on Hadamard.

5. Texture recognition based on Walsh