Applications of Linear Filtering and Local Diffusion

Processing for the analysis and enhancement of the images

using MATLAB

Author

Jimin Kim

Abstract

Development of imaging science has had a great impact on modern day science and

technology for the last few decades. By introducing different types of analysis and processing

techniques, imaging science is now widely used throughout the scientific fields such as

physics and biology but also throughout the industrial fields such as computer graphics. The

working principles of different types of imaging science techniques are based on a rigorous

mathematical framework that handles the image data. By understanding this framework and

implementing into MATLAB, one can successfully obtain the information of interest and

achieve desired results from the image analysis and processing.

Introduction/Overview

The real life example where the imaging techniques will be applied is follwing: GQ

is doing a feature article on VH1’s three time male model of the year Derek Zoolander. Much

to every beautiful person’s dismay, the ridiculously good looking photos of Derek have

created a crisis. First, the ugly protestors have corrupted Derek’s touch-upped photos by

adding heavy noise to each pixel. The only remaining photos are when Derek had a rash near

his nose. The goal here is to save Derek’s male modeling career by analyzing and improving

these photos with imaging processing techniques. For the pictures that were corrupted by the

ugly protestors, Linear Filtering technique will be used for de-noising. For the ones with a

rash near his nose, Local Diffusion technique will be used to process the bad pixels in the

region of interest. The key idea here is to understand the mathematical framework behind

these techniques and achieve the best results by preserving the reasonable image quality after

processing.

Theoretical Background

In this section, brief introduction to the mathematical theories behind the technique

of Linear Filtering and Diffusion Processing will be presented.

Linear Filtering

The principles behind the Linear Filtering are virtually identical to those behind the

time frequency filtering using Fourier transform: both methods utilize Fourier transform to

locate the center frequency and remove the noisy frequencies by applying a filter function.

Understanding the fact that any image can be transformed into a 2D matrix by converting the

each pixel information into numbers, one can perform Fourier transform on the image to

analyze in terms of Fourier modes.

The goal of Linear Filtering technique is then to locate the center frequency of the

image in its Fourier mode domain and construct a filter with an appropriate coefficient that

will remove the frequencies that correspond to the noise in the image. Once the filter with a

reasonable coefficient value is applied, the recovered image clearly demonstrates the

significantly reduced amount of noise. However, just like in time frequency filtering, a filter

with a too large coefficient can degrade the image quality by blurring the image. In the

contrast, filter with a too small coefficient will not succeed in removing the noise since the

filter might preserve the noisy frequency modes. Figure 2 and 3 demonstrate this idea more

clearly. Notice that a filter with a small coefficient barely removes any noise in the image.

However, a filter with a large coefficient successfully cleans up the noise, but degrades the

image quality significantly by blurring the image. Therefore, it is important to test different

filter coefficients and determine the value that accomplishes both reasonable image quality

and reduction of noise.

Figure 2. The example of using a filter with a small filter coefficient. Notice that the large portion

of the noise is still present within the image.

Figure 3. The example of using a filter with a large filter coefficient. Notice that all the noise are

gone, but the image quality is significantly degraded from the blurring.

Figure 1. Left picture shows the original image before Fourier transform is applied, and the right

picture shows the image’s frequency domain after the transform. Notice the central frequency at

the center of the image

Different types of filter function can be applied to the frequency domain of the image

to kill the noise but in this paper, classical Gaussian filter will be applied. The equation of 2D

Gaussian filter reads

(1)

Here, k values correspond to the frequency domain axes x and y respectively. a and b

correspond to the location of the center frequencies and and correspond to the filter

coefficients that determine the magnitude of the filter. Thus, when one increases the absolute

values of the coefficients, one decreases the size of the filter which leads to the blurrier image

when the filter is applied and vice versa. Figure 4 shows the 2D Gaussian filter as a contour

map.

Diffusion Processing

Intimately related to filtering is the use of diffusion for image enhancement. The 2D

diffusion equation follows the following form

(2)

Where

(3)

Here, u(x,y) represents the initial image and D corresponds to the diffusion

coefficient. If one considers periodic boundary conditions, then the solution of the equation 2

can be found from the usage of Fourier transform.

(4)

Figure 4. The 3D contour map of the 2D Gaussian filter. When filter coefficient values increase,

the width of the filter decreases and vice versa.

Where is the Fourier transform of u(x,y) and is the Fourier transform of the

initial condition: the noisy image. One should be able to notice a close similarity between the

equation 1 and 4: the wave numbers (spatial frequencies) in the right side of the equation 4

decay according to a Gaussian function. This observation makes a very important conclusion:

Linear Filtering with a Gaussian filter is equivalent to a Linear Diffusion of the image for

periodic boundary conditions. Hence mathematically, the technique of filtering and diffusion

are carrying out virtually identical tasks. However, the diffusion formalism can provide a

more general framework for the image enhancement since the equation 2 can be modified as

following.

Where D(x,y) is now a spatial diffusion coefficient in terms of x and y. This enables

diffusion technique to pick particular spot on the image while leaving domains with relatively

less noise untouched. When it comes to implementing this framework for 2D into MATLAB,

the governing equation becomes.

(6)

Now, if one assumes that , then the equation 6 can be reduced to

(7)

Where A is a sparse matrix that corresponds to the numerical second derivative

operator and u is the column matrix that holds 2D matrix information. Reshaping the 2D

matrix into 1D matrix for u is crucial in MATLAB implementation so that when multiplied

with A, it gives the correct result. When reshaped, each index in column matrix u is defined

as

(8)

Algorithm Implementation/Development

The algorithm implementation for both Linear Filtering and Diffusion will be

presented. The implementation of Linear Filtering in MATLAB takes a following sequence of

procedure.

Linear Filtering

1. Load the image

First, load the image to be analyzed and processed into MATLAB. This can be done

by using the built in image read function in MATLAB. Notice that once converted, a

black and white image is a 2D matrix with the row and column corresponding to the

resolution of the picture. But also note that the color image will be converted into a

3D matrix with three layers of 2D matrix that each corresponds to red, green and blue,

i,e, the RGB format.

2. Transform the matrix into ‘double’

The next step is to convert the image matrix from the uint8 format to the double so

that each element in the matrix is converted into a number. Once the matrix is

converted, one is ready to perform mathematical tasks on the image.

3. Locate the center frequency of the image

Now the image is ready to be analyzed, one should apply Fourier transform to the

image to locate the center frequency. Make sure to use 2D Fourier transform since the

matrix is 2D. Once the image has been transformed, one should also apply fftshift so

that the plot comes out correctly. Then one can use the plot function to see the

transformed image and identify the center frequency..

4. Define a frequency domain

Once the center frequency of the image is identified, one should define a frequency

domain where the Gaussian filter will be built upon. The frequency domain should

also be 2D as well with kx corresponding to the number of columns and ky

corresponding to the number of rows. Once kx and ky are defined, construct a

meshgrid (kx,ky) which will be used as a domain for Gaussian filter.

5. Construct a Gaussian filter

Now the frequency domain has been defined, one should construct a Gaussian filter

with appropriate parameters. Gaussian filter should take the form of equation 1 with

parameters a and b each corresponding to the x and y coordinates of the center

frequency. Also, define the appropriate filter coefficients so that the filter doesn’t over

filter or under filter the image.

6. Apply the filter to the frequency domain

Once the filter is defined, only thing left is to apply on the frequency matrix of the

image. Simply multiply the frequency matrix with the filter since they are both in

same dimension. One can plot the result afterward to see the filtered frequency data.

7. Recover the image

After filter is applied, and the noise components have been cleared up, one can use

the inverse Fourier transform to recover the image back to its double matrix. Don’t

forget to use 2D version of the inverse Fourier transform and also apply ifftshift

before performing the inverse transform. Finally, convert the double matrix back into

uint8 format so that the image is back to its original format.

8. Comment about filtering RGB images (Optional)

As mentioned earlier, RGB images are converted into 3D matrices since they contain

three layers of RGB colors. There are different ways of filtering these matrices but

one way is to separate each layer of color from the 3D matrix and apply the filter

independently on each layer of 2D matrix. I,E, repeat from the step 3 to 7 on each

layer of color and merge them into a single 3D matrix at the end. This way, one can

filter all three layers of colors and obtain a filtered colored image after merging.

Local Diffusion Processing

The implementation of Local Diffusion Processing is presented in this section. The

goal here is to apply the technique to only the region where the bad pixels are located.

1. Load the image

This is the same as the step 1 in Linear Filtering. Load the image to be processed so

that MATLAB can work with its matrix.

2. Transform the matrix into ‘double’

This is again the same procedure as step 2 in Linear Filtering section. Convert the

matrix into double format so that the matrix is ready to be analyzed.

3. Locate the region which the diffusion processing will be applied

Once the matrix is ready for the analysis, a region of the interest in the image should

be cropped out so that the local diffusion processing can be applied. One way to do

this is to open the image file and manually locate the region. The key idea here is to

select the region to be slightly larger than the actual size of the region to be processed.

Usually, extra 5 pixels on both left and right for the x axis and extra 5 pixels on up

and down for the y axis should suffice. Thus, the cropped region should fully contain

the part where the diffusion should be applied. This is a very crucial process because

it gives enough room to crop out the black boundaries at the edge after the diffusion

processing is applied to the region. More detail is given at step 9.

4. Define second differential operator

Once the local region of interest has been cropped out for the processing, one is ready

to define a second derivative operator that corresponds matrix A in equation 7. First

define the linear space in both x and y directions that are in same size as the cropped

matrix obtained from step 3. Next, define dx and dy for numerical differentiation

process. Next construct the column matrices with ones as their elements. Their length

should be the length of row and column of the cropped matrix. Then use spdiags and

eye function of MATLAB to construct the sparse matrix A. Make sure to use kron

function at last to create a 2D matrix version of the second derivative operator.

5. Determine the time span and diffusion coefficient

Now the second derivative operator is defined, one should determine the time span

and diffusion coefficient. The principles are similar from the Linear Filtering: larger

the time span, longer the diffusion process will take place, leading to a blurrier image.

Hence, it is important to test with many values to find a number that achieves both

reasonable image quality and removal of the bad pixels. 0.002 is a good number to

start with and one can check its affect on image quality by slowly increasing the

value. Also, set the diffusion coefficient value to the reasonable amount so that image

change is noticeable when the diffusion is applied. If the diffusion coefficient value is

too small, the change in each element of the matrix will be small as well and when

converted back into uint8 format, the region might have not been processed at all

since uint8 rounds up the decimal digits, making each element into integer value.

Therefore, if one finds no difference in the image after processing, make sure if the

diffusion coefficient value is not set too small.

6. Transform the 2D matrix into column matrix

Now the tools for the diffusion processing have been established, one should convert

the cropped matrix into a 1D matrix so that the matrix is in the form of equation 8.

This is so that MATLAB’s built in linear differential equation solver can be applied

correctly with the operator matrix A defined in step 4.

7. Define the diffusion function.

Once all the components of equation 7 are constructed, one should define a function

in a separate script to put them all together to form a set of differential equations.

Simply define a function with parameters time, initial matrix, second derivative

operator and diffusion coefficient, and let the function be in a form of D*L*u to

correspond to the equation 7.

8. Solve DEQ using ODE113

Now only thing left is to solve the linear differential equations in the form of equation

7. Use MATLAB’s built in DEQ solver ODE113 to solve the differential equations.

Simply call the diffusion function defined in step 7 with its parameters and the

solution should be in the format such that each row corresponds to the processed

image at given time span.

9. Extract the spot of interest and patch it into the original image

Now the diffusion process has been applied to the cropped region, one should crop

the matrix once again to avoid the black boundaries formed around the edge after

processing. Simply crop the original region of interest by subtracting the extra pixels

that have been added in both x axis and y axis. Finally, replace the region of interest

of the original image with the one that is processed. Display the image to see if the

patched part looks natural enough and modify the time span and diffusion coefficient

for further modifications.

9. Comment about filtering RGB images (Optional)

As it was mentioned in Linear Filtering section, if one is trying to apply the local

diffusion process for color images, one can simply repeat this procedure for each

RGB layer of color. Simply crop out the region of interest and separate into 3 layers

of matrix. Apply the diffusion process for each layer and merge them at the end. Once

merged, the remaining tasks are to crop out the black boundaries and patch it into the

original image.

Computation results/Analysis

After following the procedures of Linear Filtering in the previous section, one can

obtain the de-noised images of the Zoolander both in B/W and color. The figure 5 shows the

image enhancement that has been done by the Gaussian filtering. Notice that for both B/W

and color, the image quality is significantly improved after Linear Filtering has been applied.

For removing the rash near Zoolander’s nose, one can obtain images in figure 6 after

following the procedures for Local Diffusion process. Notice that the patched region fits

naturally in both B/W and color images.

Summary

By applying the different types of image analysis and processing techniques to the

corrupted images, one can successfully analyze and significantly enhance their overall quality.

In this paper, Linear Filtering and Local Diffusion techniques have been applied to save

Zoolander’s career as a male model. Linear Filtering technique has been applied to de-noise

the noisy images of Zoolander, and Local Diffusion technique has been applied to remove the

rash near Zoolander’s nose.

Figure 5. The left picture is the original image before filtering is applied and the right picture is

the filtered image with Gaussian filter. Notice that all the noise has been cleaned up but the image

still preserve reasonable quality.

The techniques have been applied on both B/W and color images and their

implementation procedures have been presented with step by step manner. Also,

mathematical theories behind the two techniques have been briefly discussed to understand

their mechanisms. By carrying out these tasks, one can not only understand the mathematical

framework behind the image analysis and processing, but also investigate their mechanisms

in actions by applying them on the realistic images.

Appendix A

In this section, the MATLAB functions that have been used for

implementation/development are introduced with brief implementation explanations.

imread: This function is used to read the image files into the MATLAB.

double: This function is used to convert the uint8 format into the double format.

fft2: This function is used to perform 2D Fourier transform to the image matrix.

fftshift: This function is used to shift the frequency domain for correct plotting.

Figure 6. The left picture for each figure is the original image before the local diffusion processing

is applied and the right picture is the image after fixing the rash with the diffusion processing.

Notice that right values of time span and diffusion coefficient have been applied to make the

patched spot to fit nicely into the original images

meshgrid: This function is used to create a frequency domain for the Gaussian filter.

ifft2: This function is used to perform the 2D inverse Fourier transform

ifftshift: This function is used to inverse shift the frequency domain before recovering to

image.

uint8: This function is used to convert double matrix into original uint8 matrix.

subplot: This function is used to plot the image matrix.

zeros: This function is used to create a zero matrix where all the layers of RGB are merged at

the end of the filtering.

size: This function is used to define the size of the region to be patched.

linspace: This function is used to construct the 2D second derivative operator for the

diffusion processing.

ones: This function is used to construct the axis for the 2D second derivative operator.

spdiags: This function is used to extract the nonzero diagonal elements for the 2D second

derivative operator.

eye: This function is used to return an identity matrix for constructing the 2D second

derivative operator.

kron: This function is used to return a Kronecker tensor product of matrices used for

constructing the 2D second derivative operator.

reshape: This function is used to reshape the 2D matrix into the 1D matrix before applying

ODE113.

ode113: This function is used to solve the system of linear differential equations for diffusion

processing.

function: This is used to define the diffusion function to be called by ODE113.

Appendix B

In this section, the coding for algorithms that are mentioned in this paper is presented.

Gaussian Filtering B/W

clear all; close all; clc;

A=imread('derek2','jpeg'); AA=double(A);

A1t=fft2(AA); A1ts=fftshift(A1t); %subplot(1,2,2); pcolor(log(abs(A1ts))); shading interp; kx=1:361; ky=1:253; [Kx,Ky]=meshgrid(kx,ky); F=exp(-0.0005*(Kx-181).^2-0.0005*(Ky-127).^2);

Filtered1=A1ts.*F;

%pcolor(log(abs(Filtered1))); shading interp

FixedBW=ifft2(ifftshift(Filtered1));

FixedDerek=uint8(FixedBW);

subplot(1,2,1); imshow(A); title('Before filtering','fontsize',25); subplot(1,2,2); imshow(FixedDerek); title('After filtering','fontsize',25);

Gaussian Filtering RGB

clear all; close all; clc;

A=imread('derek1','jpeg'); AA=double(A);

A1=AA(:,:,1); A2=AA(:,:,2); A3=AA(:,:,3);

A1t=fft2(A1); A1ts=fftshift(A1t); %subplot(2,2,1), image(uint8(A)); %subplot(1,3,1), pcolor(log(abs(A1ts))); shading interp

A2t=fft2(A2); A2ts=fftshift(A2t); %subplot(1,3,2), pcolor(log(abs(A2ts))); shading interp

A3t=fft2(A3); A3ts=fftshift(A3t); %subplot(1,3,3), pcolor(log(abs(A3ts))); shading interp

kx=1:361; ky=1:253; [Kx,Ky]=meshgrid(kx,ky); F=exp(-0.0005*(Kx-181).^2-0.0005*(Ky-127).^2);

Filtered1=A1ts.*F; Filtered2=A2ts.*F; Filtered3=A3ts.*F;

%subplot(1,3,1); pcolor(log(abs(Filtered1))); shading interp %subplot(1,3,2); pcolor(log(abs(Filtered2))); shading interp %subplot(1,3,3); pcolor(log(abs(Filtered3))); shading interp

Fixed1=ifft2(ifftshift(Filtered1)); Fixed2=ifft2(ifftshift(Filtered2)); Fixed3=ifft2(ifftshift(Filtered3));

FixedDerek=zeros(size(A));

FixedDerek(:,:,1)=Fixed1; FixedDerek(:,:,2)=Fixed2; FixedDerek(:,:,3)=Fixed3; FixedDerekk=uint8(FixedDerek);

subplot(1,2,1); imshow(A); title('Before filtering','fontsize',25); subplot(1,2,2); imshow(FixedDerekk); title('After filtering','fontsize',25);

Local Diffusion Processing B/W

clear all; close all; clc;

A=imread('derek4','jpeg');

AA=double(A); spot=AA(135:165,155:185);

[nx,ny]=size(spot);

x=linspace(0,1,nx); y=linspace(0,1,ny); dx=x(2)-x(1); dy=y(2)-y(1); onex=ones(nx,1); oney=ones(ny,1); Dx=(spdiags([onex -2*onex onex],[-1 0 1],nx,nx))/dx^2; Ix=eye(nx); Dy=(spdiags([oney -2*oney oney],[-1 0 1],ny,ny))/dy^2; Iy=eye(ny); L=kron(Iy,Dx)+kron(Dy,Ix);

tspan=0.006; D=0.25; spotD=reshape(spot,nx*ny,1);

[t,usol]=ode113('image_rhs',tspan,spotD,[],L,D); for j=1:length(t) spot_clean=uint8(reshape(usol(j,:),nx,ny)); end

spot_clean=spot_clean(5:25,5:25); AA(140:160,160:180,:)=spot_clean;

subplot(1,2,1); imshow(A); title('Before diffusion','fontsize',25); subplot(1,2,2); imshow(uint8(AA)); title('After diffusion','fontsize',25);

Local Diffusion Processing RGB

clear all; close all; clc;

A=imread('derek3','jpeg');

AA=double(A);

A1=AA(:,:,1); spot1=A1(135:165,155:185); A2=AA(:,:,2); spot2=A2(135:165,155:185); A3=AA(:,:,3); spot3=A3(135:165,155:185);

spot=zeros(31,31,3); spot(:,:,1)=spot1; spot(:,:,2)=spot2; spot(:,:,3)=spot3;

[nx,ny]=size(spot1);

x=linspace(0,1,nx); y=linspace(0,1,ny); dx=x(2)-x(1); dy=y(2)-y(1); onex=ones(nx,1); oney=ones(ny,1); Dx=(spdiags([onex -2*onex onex],[-1 0 1],nx,nx))/dx^2; Ix=eye(nx); Dy=(spdiags([oney -2*oney oney],[-1 0 1],ny,ny))/dy^2; Iy=eye(ny); L=kron(Iy,Dx)+kron(Dy,Ix);

tspan=0.006; D=0.25; spot1D=reshape(spot1,nx*ny,1); spot2D=reshape(spot2,nx*ny,1); spot3D=reshape(spot3,nx*ny,1);

[t,usol]=ode113('image_rhs',tspan,spot1D,[],L,D); for j=1:length(t) spot1_clean=uint8(reshape(usol(j,:),nx,ny)); end

[t,usol]=ode113('image_rhs',tspan,spot2D,[],L,D); for j=1:length(t) spot2_clean=uint8(reshape(usol(j,:),nx,ny)); end

[t,usol]=ode113('image_rhs',tspan,spot3D,[],L,D); for j=1:length(t) spot3_clean=uint8(reshape(usol(j,:),nx,ny)); end

spot_clean=zeros(31,31,3); spot_clean(:,:,1)=spot1_clean; spot_clean(:,:,2)=spot2_clean; spot_clean(:,:,3)=spot3_clean;

%subplot(3,1,1); image(uint8(spot)); %subplot(3,1,2); image(uint8(spot_clean));

spot_clean=spot_clean(5:25,5:25,:); AA(140:160,160:180,:)=spot_clean;

subplot(1,2,1); imshow(A); title('Before diffusion','fontsize',25); subplot(1,2,2); imshow(uint8(AA)); title('Before diffusion','fontsize',25);

Diffusion Function

function rhs=image_rhs(t,u,dummy,L,D) rhs=D*L*u; end