CS4501:IntroductiontoComputerVision

Filtering,Frequency,andEdges

Various slides from previous courses by: D.A.Forsyth(Berkeley/UIUC),I.Kokkinos(Ecole Centrale/UCL).S.Lazebnik (UNC/UIUC),S.Seitz(MSR/Facebook),J.Hays(Brown/GeorgiaTech),A.Berg(StonyBrook/UNC),D.Samaras(StonyBrook).J.M.Frahm (UNC),V.Ordonez(UVA).

• ImageProcessing:Brightness• ImageFiltering:MeanFilter• ImageBlurring• ImageGradients:TheSobelOperator

LastClass

• RecaponConvolutionalOperations• ImageGradients:TheSobelOperator• FrequencyDomain• FilteringinFrequency• GoogleColaboratory

Today’sClass

What we see What a computer seesSource: S. Narasimhan

ImagesasMatrices

ImagesasFunctionsz = 𝑓(𝑥, 𝑦)

• Thedomainofxandyis[0,img-width)and[0andimg-height)• x,andyarediscretizedintointegervalues.

BasicImageProcessing

𝐼 𝛼𝐼

𝛼 > 1

BasicImageProcessing

𝐼 𝛼𝐼

0 < 𝛼 < 1

Imagefiltering:Convolutionoperator

ImageCredit:http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

𝑘(𝑥, 𝑦)

𝑔 𝑥, 𝑦 =11𝑘 𝑢, 𝑣 𝑓(𝑥 − 𝑢, 𝑦 − 𝑣)�

7

�

8

Image filtering: Convolution operatore.g.meanfilter

ImageCredit:http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

𝑘(𝑥, 𝑦)

𝑘(𝑥, 𝑦) =

1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10 20 30 30 30 20 10

0 20 40 60 60 60 40 20

0 30 60 90 90 90 60 30

0 30 50 80 80 90 60 30

0 30 50 80 80 90 60 30

0 20 30 50 50 60 40 20

10 20 30 30 30 30 20 10

10 10 10 0 0 0 0 0

[.,.]h[.,.]f

Imagefiltering111111111],[g ××

Credit: S. Seitz

],[],[],[,

lnkmflkgnmhlk

++=å

Image filtering: Convolution operatorImportantfilter:gaussian filter(gaussian blur)

ImageCredit:http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

𝑘(𝑥, 𝑦)

𝑘(𝑥, 𝑦) =

1/16 1/8 1/16

1/8 1/4 1/8

1/16 1/8 1/16

Image filtering: Convolution operatore.g.gaussian filter(gaussian blur)

ImageCredit:http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

SharpeningFilter

Original

111111111

000020000 -

Sharpening filter- Accentuates differences with local average

Source: D. Lowe

SharpeningFilter

Source: D. Lowe

Image filtering: Convolution operatorImportantFilter:Sobeloperator

ImageCredit:http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

𝑘(𝑥, 𝑦)

𝑘(𝑥, 𝑦) =

1 0 -1

2 0 -2

1 0 -1

SobelinX

-101

-202

-101

Vertical Edge(absolute value)

Sobel

SlidebyJamesHays

SobelinY

-1-2-1

000

121

Horizontal Edge(absolute value)

Sobel

SlidebyJamesHays

Sobeloperatorsareequivalentto2Dpartialderivativesoftheimage• Verticalsobel operator– PartialderivativeinX(width)• Horizontalsobel operator– PartialderivativeinY(height)

• Cancomputemagnitudeandphaseateachlocation.

• Usefulfordetectingedges

https://en.wikipedia.org/wiki/Sobel_operator

Sobelfiltersare(approximate)partialderivativesoftheimage

𝜕𝑓(𝑥, 𝑦)𝜕𝑥

= lim=→?

𝑓 𝑥 + ℎ, 𝑦 − 𝑓(𝑥, 𝑦)ℎ

𝑓(𝑥, 𝑦) beyourinputimage,thenthepartialderivativeis:Let

𝜕𝑓(𝑥, 𝑦)𝜕𝑥

= lim=→?

𝑓 𝑥 + ℎ, 𝑦 − 𝑓(𝑥 − ℎ, 𝑦)2ℎ

Also:

Butdigitalimagesarenotcontinuous,theyarediscrete

ΔD𝑓[𝑥, 𝑦] = 𝑓[𝑥 + 1, 𝑦] − 𝑓[𝑥, 𝑦]

𝑓[𝑥, 𝑦] beyourinputimage,thenthepartialderivativeis:Let

ΔD𝑓[𝑥, 𝑦] = 𝑓[𝑥 + 1, 𝑦] − 𝑓[𝑥 − 1, 𝑦]Also:

Butdigitalimagesarenotcontinuous,theyarediscrete

ΔD𝑓[𝑥, 𝑦] = 𝑓[𝑥 + 1, 𝑦] − 𝑓[𝑥, 𝑦]

𝑓[𝑥, 𝑦] beyourinputimage,thenthepartialderivativeis:Let

ΔD𝑓[𝑥, 𝑦] = 𝑓[𝑥 + 1, 𝑦] − 𝑓[𝑥 − 1, 𝑦]Also:

-1 1

-1 0 1

k(x,y)=

k(x,y)=

SobelOperatorsSmoothinYandthenDifferentiateinX

1 0 -1k(x,y)=

1

2

1

* =

1 0 -1

2 0 -2

1 0 -1

SimilarlytodifferentiateinY

Frequency

FigurebyNationalInstruments

Anyfunctioncanbeapproximatedbyapolynomialfunction

…ifyouletyourpolynomialhaveahighdegree

…ANDyoucancomputethederivativesoftheoriginalfunctioneasily.

TaylorSeriesexpansion

Difficultinpractice

https://brilliant.org/wiki/taylor-series-approximation/

JeanBaptiste JosephFourier(1768-1830)hadcrazyidea(1807):Any univariate functioncanberewrittenasaweightedsumofsines andcosinesofdifferentfrequencies.

• Don’tbelieveit?– NeitherdidLagrange,Laplace,Poissonandotherbigwigs

– NottranslatedintoEnglishuntil1878!

• Butit’s(mostly)true!– calledFourierSeries– therearesomesubtlerestrictions

...themannerinwhichtheauthorarrivesattheseequationsisnotexemptofdifficultiesand...his

analysistointegratethemstillleavessomethingtobedesiredonthescoreofgeneralityandeven

rigour.

Laplace

LagrangeLegendre

SlidebyJamesHays

Example

SlidebyEmmanuelAgu

DiscreteFourierTransform

𝐹 𝑢 = 1 𝑓 𝑥 exp −2𝜋𝑖𝑥𝑢𝑁

NOP

DQ?

𝐹 𝑢 = 1 𝑓 𝑥 cos −2𝜋𝑥𝑢𝑁

+ 𝑖 sin −2𝜋𝑥𝑢𝑁

NOP

DQ?

InverseDiscreteFourierTransform

𝐹 𝑢 = 1 𝑓 𝑥 exp −2𝜋𝑖𝑥𝑢𝑁

NOP

DQ?

𝑓 𝑥 =1𝑁1 𝐹 𝑢 exp 2𝜋𝑖

𝑥𝑢𝑁

NOP

7Q?

DiscreteFourierTransform

Moregenerallyforimages(2DDFTandiDFT)

𝐹 𝑢, 𝑣 = 1 1 𝑓 𝑥, 𝑦 exp −2𝜋𝑖𝑥𝑢𝑀+𝑦𝑣𝑁

NOP

WQ?

XOP

DQ?

𝑓 𝑥, 𝑦 =1𝑀𝑁

1 1𝐹 𝑢, 𝑣 exp 2𝜋𝑖𝑥𝑢𝑀+𝑦𝑣𝑁

NOP

8Q?

XOP

7Q?

DiscreteFourierTransform- Visualization

SlidebyA.Zisserman

FourierTransform

• Fouriertransformstoresthemagnitudeandphaseateachfrequency• Magnitudeencodeshowmuchsignalthereisataparticularfrequency• Phaseencodesspatialinformation(indirectly)• Formathematicalconvenience,thisisoftennotatedintermsofrealandcomplexnumbers

22 )()( ww IRA +±=)()(tan 1

wwf

RI-=Amplitude: Phase:

SlidebyJamesHays

ImageFilteringintheFrequencyDomain

SlidebyA.Zisserman

ImageFilteringintheFrequencyDomain

SlidebyA.Zisserman

ImageFilteringintheFrequencyDomain

SlidebyA.Zisserman

TheConvolutionTheorem• TheFouriertransformoftheconvolutionoftwofunctionsistheproductoftheirFouriertransforms

• Convolution inspatialdomainisequivalenttomultiplication infrequencydomain!

]F[]F[]F[ hghg =*

]]F[][F[F* 1 hghg -=

Howcanthisbeuseful?

Questions?

39