Introduction to Visual Computing

Prof. Feng Liu

Winter 2020

http://www.cs.pdx.edu/~fliu/courses/cs410/

01/09/2020

Last Time

Course overview

◼ Admin. Info

◼ Computer Vision

Computer Vision at PSU

Image representation

Color

2

Today

Filter

3

Today

Filters and its applications

4

naïve denoising

Gaussian blur

better denoising

edge-preserving filter

Slide credit: Sylvain Paris and Frédo Durand

noisy image

Image filtering: compute function of local neighborhood at each pixel position

One type of “Local operator,” “Neighborhood operator,” “Window operator”

Useful for:◼ Enhancing images

Noise reduction, smooth, resize, increase contrast, etc.

◼ Extracting information from images

Texture, edges, distinctive points, etc.

◼ Detecting patterns

Template matching, e.g., eye template

Slide credit: C. Dyer and D. Hoiem

Image Filtering

111

111

111

Slide credit: David Lowe

],[g

Example: Box Filter

Image Correlation Filtering

Center filter g at each pixel in image f

Multiply weights by corresponding pixels and

add them together

Set resulting value in output image h

g is called a filter, mask, kernel, or template

Linear filtering is sum of dot product at each

pixel position

Filtering operation called cross-correlation,

and denoted h = f g

Slide credit: C. Dyer

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Credit: S. Seitz

],[],[],[,

lnkmflkgnmhlk

++=

[.,.]h[.,.]f

Image Filtering111

111

111

],[g

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]f

Image Filtering111

111

111

],[g

],[],[],[,

lnkmflkgnmhlk

++=Credit: S. Seitz

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10 20

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]f

Image Filtering111

111

111

],[g

Credit: S. Seitz

],[],[],[,

lnkmflkgnmhlk

++=

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10 20 30

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]f

Image Filtering111

111

111

],[g

],[],[],[,

lnkmflkgnmhlk

++=Credit: S. Seitz

0 10 20 30 30

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]f

Image Filtering111

111

111

],[g

],[],[],[,

lnkmflkgnmhlk

++=Credit: S. Seitz

0 10 20 30 30

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]f

Image Filtering111

111

111

],[g

?

],[],[],[,

lnkmflkgnmhlk

++=Credit: S. Seitz

0 10 20 30 30

50

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]f

Image Filtering111

111

111

],[g

?

],[],[],[,

lnkmflkgnmhlk

++=Credit: S. Seitz

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10 20 30 30 30 20 10

0 20 40 60 60 60 40 20

0 30 60 90 90 90 60 30

0 30 50 80 80 90 60 30

0 30 50 80 80 90 60 30

0 20 30 50 50 60 40 20

10 20 30 30 30 30 20 10

10 10 10 0 0 0 0 0

[.,.]h[.,.]f

Image Filtering 111

111

111],[g

],[],[],[,

lnkmflkgnmhlk

++=Credit: S. Seitz

What does it do?

• Replaces each pixel with an average of its neighborhood

• Achieves smoothing effect (i.e., removes sharp features)

• Weaknesses:

• Blocky results

• Axis-aligned streaks

111

111

111

Slide credit: David Lowe

],[g

Box Filter

Smoothing with Box Filter

Slide credit: C. Dyer

Properties of Smoothing Filters

Smoothing

◼ Values all positive

◼ Sum to 1 constant regions same as input

◼ Amount of smoothing proportional to mask size

◼ Removes “high-frequency” components

◼ “low-pass” filter

Slide credit: C. Dyer

000

010

000

Original

?

Source: D. Lowe

Practice with Linear Filters

000

010

000

Original Filtered

(no change)

Source: D. Lowe

Practice with Linear Filters

000

100

000

Original

?

Source: D. Lowe

Practice with Linear Filters

000

100

000

Original Shifted left

by 1 pixel

Source: D. Lowe

Practice with Linear Filters

Original

111

111

111

000

020

000

- ?

(Note that filter sums to 1)

Source: D. Lowe

Practice with Linear Filters

Original

111

111

111

000

010

000

- ?

(Note that filter sums to 1)

Source: D. Lowe

Practice with Linear Filters

000

010

000

+

Original

111

111

111

000

020

000

-

Sharpening filter- Sharpen an out of focus image by subtracting a multiple of a blurred version

Source: D. Lowe

Practice with Linear Filters

Source: D. Lowe

Sharpening

h = f + k(f * g) where k is a small positive constant and g =

Called unsharp masking in photography

0 1 0

1 -4 1

0 1 0

called a Laplacian mask

Sharpening by Unsharp Masking

Sharpening using Unsharp Mask Filter

Original Filtered result

Slide credit: C. Dyer

-101

-202

-101

Vertical Edge(absolute value)

Sobel

Other Filters: Edge Detection

Slide credit: C. Dyer

-1-2-1

000

121

Horizontal Edge(absolute value)

Sobel

Other Filters: Edge Detection

Slide credit: C. Dyer

Linearity:

filter(f1 + f2) = filter(f1) + filter(f2)

Shift invariance: same behavior regardless of pixel

location

filter(shift(f)) = shift(filter(f))

Source: S. Lazebnik

Key Properties of Linear Filters

Weight contributions of neighboring pixels by nearness

Constant factor at front makes volume sum to 1

Convolve each row of image with 1D kernel to produce new

image; then convolve each column of new image with same 1D

kernel to yield output image

0.003 0.013 0.022 0.013 0.0030.013 0.059 0.097 0.059 0.0130.022 0.097 0.159 0.097 0.0220.013 0.059 0.097 0.059 0.0130.003 0.013 0.022 0.013 0.003

5 x 5, = 1

Slide credit: Christopher Rasmussen

Gaussian Filtering

Smoothing with a box

actually doesn’t compare at

all well with a defocused lens

Most obvious difference is

that a single point of light

viewed in a defocused lens

looks like a fuzzy blob; but

the averaging process would

give a little square

Gaussian is isotropic (i.e.,

rotationally symmetric) A Gaussian gives a good

model of a fuzzy blob

It closely models many

physical processes (the sum

of many small effects)

Slide by D.A. Forsyth

Smoothing with a Gaussian

What does Blurring take away?

originalSlide credit: C. Dyer

What does Blurring take away?

smoothed (5x5 Gaussian)Slide credit: C. Dyer

Smoothing with Gaussian Filter

Smoothing with Box Filter

input

Slide by S. Paris

box average

Slide by S. Paris

Gaussian blur

Slide by S. Paris

What parameters matter here?

Standard deviation, , of Gaussian: determines extent

of smoothing

σ = 2 with 30 x 30 kernel

σ = 5 with 30 x 30 kernel

Source: D. Hoiem

Gaussian Filters

Slide credit: C. Dyer

…

Parameter σ is the “scale” / “width” / “spread” of the Gaussian kernel, and controls the amount of smoothing

Smoothing with a Gaussian

Slide credit: C. Dyer

Slide by D.A. Forsyth

Small

Slide by D.A. Forsyth

Medium

Slide by D.A. Forsyth

Large

Gaussian filters

= 30 pixels= 1 pixel = 5 pixels = 10 pixels

Slide credit: C. Dyer

Source: K. Grauman

* =

Cascading Gaussian Filters

Remove “high-frequency” components from the image

(low-pass filter)

◼ Images become smoother / more blurred

Convolution with self is another Gaussian

◼ Convolving two times with Gaussian kernel of width σ is

same as convolving once with kernel of width σ√2

A function g(x, y) is separable if g(x, y) = g1(x) g2(y)

The Gaussian function is separable:

e -[(x2 + y2)/22] = e -[x

2/22]* e -[y

2/22]

First convolve the image with a 1D horizontal filter

Then convolve the result of the first convolution with a 1D vertical filter

For a k x k Gaussian filter, 2D convolution requires k2 multiplications and k2-1 additions per pixel

But using the separable filters, we reduce this to 2kmultiplications and 2k-1 additions per pixel

Separability

Slide credit: C. Dyer

Separability

2 3 3

3 5 5

4 4 6

1 2 1

1

2

1

18

11

18

18

11

18

65

1 2 11

2

1

1 2 1

2 4 2

1 2 1

2 3 3

3 5 5

4 4 6

= 2 + 6 + 3 = 11

= 6 + 20 + 10 = 36

= 4 + 8 + 6 = 18

65

=

x

g1 =

g2 =

f =

Slide credit: C. Dyer

What parameters matter here?

Size of kernel or mask

σ = 5 with 10 x 10 kernel

σ = 5 with 30 x 30 kernel

Gaussian Filters

Slide credit: C. Dyer

Gaussian function has infinite “support” but need a finite-size kernel

Values at edges should be near 0

98.8% of area under Gaussian in mask of size 5σ x 5σ

In practice, use mask of size 2k+1 x 2k+1 where k 3

Normalize output by dividing by sum of all weights

How big should the filter be?

Slide credit: C. Dyer

What about near the edge?

◼ the filter window falls off the edge of the image

◼ need to extrapolate

◼ methods:

clip filter (black)

wrap around

copy edge

reflect across edge

Source: S. Marschner

Practical Matters

Filter Re-cap

56

naïve denoising

Gaussian blur

better denoising

edge-preserving filter

Slide credit: Sylvain Paris and Frédo Durand

noisy image

Median Filter

• Replace pixel by the

median value of its

neighbors

• No new pixel values

introduced

• Removes spikes: good

for impulse, salt &

pepper noise

• Nonlinear filter

Slide credit: C. Dyer

Salt and pepper noise

Median filtered

Plots of a row of the image

Median Filter

Slide credit: M. Hebert, C. Dyer

Median filter is edge preserving

Median Filter

Slide credit: C. Dyer

Slide credit: C. Dyer

Median Filter (19 x 19 window)

input outputimages by J. PlushSlide credit: C. Dyer

Next Time

More filters

Feature detection

62