computer vision and graphics (ee2031)info.ee.surrey.ac.uk/teaching/.../imageprocessing... · •...

Computer Vision and Graphics (ee2031) Digital Image Processing II

Dr John Collomosse J.Collomosse@surrey.ac.uk

Centre for Vision, Speech and Signal Processing University of Surrey

Learning Outcomes After attending this lecture, and doing the reading and labwork, you should be able to:

•  Recommend appropriate filtering techniques to enhance images under different types of image noise.

•  Describe the phenomenon of image edges, and implement common edge detection approaches using linear filtering.

•  Describe the use of basic image histogram operators to reveal hidden visual detail within an image.

•  Explain aliasing and it’s relationship to Nyquist’s limit. Be able to discuss the implications for 2D and 3D graphics.

•  Describe image interpolation, and its link to linear filtering and convolution.

Credit: Some images in these slides from Noah Snavely (Cornell). David Lowe (Columbia). Steve Seitz (Washington). Various creative commons sources incl. Wikipedia.

Recall: Convolution

Consider a window containing a set of values.

For each pixel in the input:

1. Window values are multiplied with image beneath

2. The sum of these products is written to output image.

e.g. (0 x 1) + (4 x 1) + (0 x 1) +... = 4/9

0 4 0 0 0 0

0 0 0 0 0 0

0 0 0 3 0 0

0 0 0 0 0 0

0 2 0 0 0 0

0 0 0 0 0 0

0 1 0 0

0 0 0 0

0 1 0 0

0 0 0 0

1 1 1

1 1 1

1 1 1

1/9 x

Convolution

Input f(x,y) Output g(x,y)

Convolution in detail: 1D

Consider an input signal f(x),

Transformed by filter h(x),

Into an output signal g(x).

Impulse

Gaussian

* =  f(x) h(x) g(x)

[k = ½ width of filter] Continuous Discrete

Convolution in detail: 2D (Discrete)

0 4 0 0 0 0

0 0 0 0 0 0

0 0 0 3 0 0

0 0 0 0 0 0

0 2 0 0 0 0

0 0 0 0 0 0

0 1 0 0

0 0 0 0

0 1 0 0

0 0 0 0

1 1 1

1 1 1

1 1 1

1/9 x =   * F(i,j) H(i,j) G(i,j)

Consider an input signal F(i,j),

Transformed by filter H(i,j),

Into an output signal G(i,j).

Quick Quiz

2.Which low-pass filter produced fewer image artifacts?

0.11  0.11  0.11  

0.11  0.11  0.11  

0.11  0.11  0.11  

0.06  0.13  0.06  

0.13  0.24  0.13  

0.06  0.13  0.06  3x3 box filter 3x3 Gaussian

1.What is the convolution theorem? Why is it useful?

3. What is the freq. domain representation of these filters?

5. Explain why one low-pass filter outperforms the other?

4. What are the implications of compact support in 2D DFT of these filters?

Noise reduction: An Alternative Linear low-pass filters (esp. Gaussian filters) are good at attenuating “Gaussian noise”

f(x,y) = I(x,y) + N(0,σ)

1 10 100 1000

image = signal + noise

Gaussian noise

=  1  pixel   =  2  pixels   =  5  pixels  

Applying a linear low-pass (Gaussian) filter

Effectiveness of linear filter on Gaussian noise (good)

Noise reduction: An Alternative Another common type of noise: “Salt and pepper” noise, e.g.

f(x,y) = I(x,y) if 0 > p > 0.94

0 if 0.94 > p > 0.97

1 if 0.97 > p > 1

Original  image   Gaussian  noise   Salt  and  pepper  noise  (each  pixel  has  some  chance  of  being  switched  to  zero  or  one)  

Salt and Pepper noise

Applying a linear low-pass (Gaussian) filter

Effectiveness of linear filter on Salt and Pepper noise (bad)

Why?

=  1  pixel   =  2  pixels   =  5  pixels  p  =  10%  

Salt and Pepper noise

Is this a linear filter? (Can we use convolution?)

Salt and pepper noise best attenuated using a Median Filter

=  1  pixel   =  2  pixels   =  5  pixels  

3x3  window   5x5  window   7x7  window  

p  =  10%  

Salt and Pepper noise - Median

Gaussian filter

Median filter

Edge Detection Linear filtering can be used to detect edges (high pass filter)

An edge is an intensity discontinuity in an image, caused by one or more physical factors:

depth discontinuity

surface color discontinuity

illumination discontinuity

surface normal discontinuity

Edge Detection Linear filtering can be used to detect edges (high pass filter)

An edge is an intensity discontinuity

Plotting intensity along this cross-section

This is a step-edge. Real image edges are “softer” than this.

Edge Detection Differentiating the image signal f(x,y), e.g. δf/δx

|δf/δx|

f(x)

f(x) Partial derivative

Edge Detection To compute derivative on a discrete signal we subtract neighbouring samples (finite difference)

f(x,y)

0  0  0  

-­‐1  0  1  

0  0  0  

-­‐1  1  -­‐1  1   -­‐1  1  

The obvious way to implement this in our 2D linear filter framework:

δf/δx

0  -­‐1  0  

0  0  0  

0  1  0  δf/δy

δf/δx

Image Gradient Filtering f(x,y) independently to get δf/δx and δf/δy gives us a vector quantity for each pixel; the image gradient...

The  gradient  points  in  the  direcIon  of  most  rapid  increase  in  intensity  

The  edge  strength  is  given  by  the  gradient  magnitude:  

The  gradient  (edge)  direcIon  is  given  by:  

Image Gradient Example of processed image:

δf/δx δf/δy

f(x,y)

Working with Real Edges

Noisy  input  image  

Working with Real Edges

f

h

f * h

To  find  edges,  look  for  peaks  in  

Smoothed edge detection Both smoothing and differentiation are possible via convolution

Convolution is associative

So combine smoothing/differentiation into single filter.

Gaussian   DerivaIve  of  Gaussian  (x)  

Gaussian Derivatives The two first-order Gaussian derivatives.

The Gaussian derivative differentiates the image in one direction (i.e. x or y) and smoothes in the orthogonal direction

x-­‐direcIon   y-­‐direcIon  

Sobel operator A common approximation to the first-order Gaussian Derivative that fits in a 3x3 window is the Sobel filter:

-­‐1  0  1  

-­‐2  0  2  

-­‐1  0  1  δf/δx

-­‐1  -­‐2  -­‐1  

0  0  0  

1  2  1  δf/δy

Sobel is frequently used in Computer Vision as a simple edge detector. A less common approximation is the Prewitt filter:

Which low-pass filter do you think Prewitt is composed with?

-­‐1  0  1  

-­‐1  0  1  

-­‐1  0  1  δf/δx

-­‐1  -­‐1  -­‐1  

0  0  0  

1  1  1  δf/δy

Sobel operator: Example

Sobel operator: Example

Smoothed edges are imprecisely localised

Global Thresholding A simple way to decide if a pixel is part of an edge or not is to threshold the edge strength field at a constant value.

This creates a “binary mask”. There are more sophisticated thresholding techniques available in the literature.

Image Histograms

Sometimes poor capture conditions can hide visual detail.

In such situations you can try manipulating the image histogram.

Image Histograms

Simple transfer functions

Original

Brightness

(+ 0.3)

Contrast

(x 1.3)

Gamma Correction

out= in γ

Original

Gamma

(0.2)

Correct for the non-linear response of a display/capture device

Localises contrast enhancement to part of histogram e.g. shadow

Gamma Correction

out= in γ Correct for the non-linear response of a display/capture device

Here, y=1/γ

Histogram Equalisation Distributing the area/energy evenly under the histogram often improves visibility of detail.

This increases the dynamic range of the image

As before, achieved via a monotonic transfer function

Histogram Equalisation

number of pixels with intensity k or less

pixels in image Number of intensity levels in image

Image Warping

How can we reduce the size (scale) this large image so it fits on the screen?

Image Sub-sampling

Throw  away  every  other  row  and  column  to  create  a  1/2  size  image  

-­‐  called  image  sub-­‐sampling  

1/4

1/8

We are ‘re-sampling’ the signal at a lower

sampling rate.

Image Warping

1/4    (2x  zoom)   1/8    (4x  zoom)  1/2  

Zoom in on the smaller images and you see artifacts that weren’t present in the larger images.

Aliasing

“Wave” artifacts appear in the subsampled version where fine detail was present in the original. This is called aliasing.

Aliasing Similar problems occur in 3D computer graphics with distant geometry or fine scale texture:

Aliasing If you sample a signal at “too low” a rate, you will get aliasing.

High frequencies “alias” as lower frequency signals

You must sample a signal using a sampling rate at least twice the highest frequency present.

This minimal sampling rate is called the Nyquist rate

The maximum frequency that can be sampled at a given rate is referred to as the Nyquist limit (i.e. = 2x Nyquist rate)

If you sample below the Nyquist rate, aliasing will occur

Anti-Aliasing We can avoid aliasing when image warping, by filtering the signal to remove frequencies above the Nyquist limit.

G 1/4

G 1/8

Gaussian 1/2

Anti-Aliasing - Comparison Sub-sampling with Gaussian pre-filtering

G 1/4 G 1/8 Gaussian 1/2

Anti-Aliasing - Comparison Sub-sampling without Gaussian pre-filtering

blur  

F0 H *

subsample   blur   subsample   …  F1

F1 H *

F2 F0

Gaussian pre-filtering

blur  

F0 H *

subsample   blur   subsample   …  F1

F1 H *

F2 F0

{ Gaussian  pyramid  

Anti-Aliasing - 3D Graphics How might we anti-alias when rendering 3D graphics? (Consider the window operation of convolution / low-pass filtering)

R. Cook – SIGGRAPH ‘86

Distributed Ray Tracing

Anti-Aliasing - 3D Graphics Two solutions:

1. Distributed Ray Tracing (Cook, SIGGRAPH 86)

2. MIP Mapping (multus in parvum)

(Williams, SIGGRAPH 83)

Upsampling How can we increase the size of this image?

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer,

Recall  a  digital  image  has  compact  support  

•  It  is  a  discrete  point-­‐sampling  of  a  conInuous  funcIon  •  If  we  could  somehow  reconstruct  the  original  funcIon,  any  new  

image  could  be  generated,  at  any  resoluIon  and  scale    

1   2   3   4   5  

d = 1 in this example

Upsampling How can we increase the size of this image?

But  we  don’t  know  f,  so  we  have  to  model  in  some  way  

Simplest  model  approximaIng  f,  is    

to  duplicate  the  closest  pixel.    

•  “Nearest  Neighbour”  interpolaIon  •  Not  really  an  “interpolaIon”  

1   2   3   4   5  

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer,

Ideal  scenario  i.e.  if  we  knew  f

Bilinear interpolation Spatial implementation:

4 known pixels (Q) at integer f(x,y)

Need to know P?

Nearest-­‐neighbor  interpolaIon   Bilinear  interpolaIon  

Upsampling How can we increase the size of this image?

1   2   3   4   5  

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer,

Ideal  scenario  i.e.  if  we  knew  f

Recall: Convolution

Consider an input signal f(x),

Transformed by filter h(x),

Into an output signal g(x).

Impulse

Gaussian

* =  f(x) h(x) g(x)

[k = ½ width of filter] Continuous Discrete

Linear Interpolation How can we increase the size of this image?

1   2   3   4   5  

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer,

Ideal  scenario  i.e.  if  we  knew  f

1   2   3   4   5  

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer,

2.5  

1  Using  triangular  or  “tent”  kernel

Bilinear interpolation The “tent” gives us a linear interpolation of surrounding values, weighting by distance.

1   2   3   4   5  

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer,

2.5  

1  Using  triangular  or  “tent”  kernel

In 2D?

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer,

performs    linear  interpolaIon  

(tent  funcIon)  performs    bilinear  interpola,on  

Image Interpolation

“Ideal”  reconstrucIon  

Nearest-­‐neighbor  interpolaIon  

Linear  interpolaIon  

Gaussian  reconstrucIon  

Image Interpolation

Nearest-­‐neighbor  interpolaIon   Bilinear  interpolaIon   Bicubic  interpolaIon  

Original  image:                    x  10  

Image Interpolation

Also  used  for  general  image  warping  

Summary After attending this lecture, and doing the reading and labwork, you should be able to:

•  Recommend appropriate filtering techniques to enhance images under different types of image noise.

•  Describe the phenomenon of image edges, and implement common edge detection approaches using linear filtering.

•  Describe the use of basic image histogram operators to reveal hidden visual detail within an image.

•  Explain aliasing and it’s relationship to Nyquist’s limit. Be able to discuss the implications for 2D and 3D graphics.

•  Describe image interpolation and its link to linear filtering and convolution.