pde-based methods for image and shape processing applications alexander belyaev school of...

PDE-based Methods for Image and Shape Processing Applications

Alexander BelyaevSchool of Engineering & Physical Sciences

Heriot-Watt University, Edinburgh

Institute of Sensors, Signals & Systems

Very active research area

+ dozens of books and thousands of research papers

Joachim Weickert, Anisotropic Diffusion in Image Processing

Tony Chan & Jianhong Shen, Image Processing and Analysis: Variational, PDE, Wavelet, and Stochastic Methods.

Guillermo Sapiro, Geometric Partial Diffrential Equations and Image Analysis.

Gilles Aubert & Pierre Kornprobst, Mathematical Problems in Image Processing.

Equations

073 23 xxx An algebraic equation

dt

tduutF

dt

tud,,

2

2

An ordinary differential equation

tyxuuy

u

x

u

t

u,,,

2

2

2

2

A partial differential equation

Usually it is not possible to solve partial differential equations (PDEs) analytically and they are solved numerically.

I. Partial Differential Equations (PDEs)

PDEs are equations involving partial derivatives of an unknown function.

For example, the so-called heat or diffusion equation is given by

yxuyxuy

u

x

u

t

u

tyxuu

,0,,

,,

0

2

2

2

2 Describes temperature distribution in a material or concentration of particles in a medium or a random walk.

Fourier transform, Gaussian smoothing, and linear diffusion

,, 22 fyxf FF

c1It explains why boosts high frequencies

yxuyxuy

u

x

u

t

u

tyxuu

,0,,

,,

0

2

2

2

2

Fourier transform w.r.t. x and y

,0,,

,,

0

22

UU

Ut

U

tUU

tUtyxu ,,,, F

A very simple ordinary differential equation. Can be easily solved analytically

I. Linear diffusion (heat/diffusion equation)

xfKtxu

xftxux

u

t

u

t

t

*,

,,

2

02

2

Proof: apply Fourier transform w.r.t. x, solve the resulting ordinary differential equation, apply inverse Fourier transform to the solution.

Thus linear diffusion is equivalent to Gaussian smoothing (convolution with Gaussian). This leads to a simple way to solve the heat equation on a plane (in space).

In practice the heat equation is usually solved numerically by using finite difference approximations or finite element methods.

I. A Brief History of PDE Methods in IP

1955

I. Kovasznay & Joseph: inverting diffusion for image sharpening purposes

I. Image enhancement/deblurring

, 1 ,I x y c I x y

2 3

12 6 !

n

c c c ce c

n

Unsharp masking (a popular image enhancement technique)

, 1 ,N

I x y c I x y

Iterated unsharp masking

I. Dennis Gabor on image enhancement

Dennis Gabor (Nobel prize in physics for inventing holography, 1971): “Information theory and electron microscopy”, 1965

Simple image sharpening

Original

111

111

111

Convolution with mask

Blurred

I. Simple image sharpening

Original

1111-81111

000010000

-

c1

Boosting high frequencies2

2

2

2

yx

Sharpened

I. Image enhancement with stabilized inverse diffusion

Can be used for deblurring Gaussian blur

tyxItyxItyxI h ,,,,pass-low,,

utu is ill posed (unstable). So a regularization is needed

A. Belyaev, ”Implicit image differentiation and filtering with applications to image sharpening.” SIAM Journal on Imaging Sciences, 6(1):660–679, 2013.

I. Stabilized inverse diffusion 2

, , implicit low-pass filtering , , , ,hI x y t dt I x y t dt I x y t

I. Stabilized inverse diffusion 3

I. Very recent use of heat (diffusion) equation

PDE: Hopf-Cole transformation

on 0,1 esapproximat 01

on 0,10

,

exp :onSubstituti

ynumericall solve easy to PDELinear :Poisson Screened

0,on 1,in 0

22

2

2

22

2

2

uuutu

uutuvvtv

x

u

t

v

x

u

t

v

x

v

x

u

t

v

x

v

txuxv

tvvtv

iiiii

eikonal equation

Hopf-Cole transformation

I. PDE: Hopf-Cole transformation

xwtxu

wwtw

xwxv

vvtv

txuxv

tuutu

1ln

on 0,in 1

1 :onSubstituti

ynumericall solve easy to PDELinear

on 1,in 0

exp :onSubstituti

1 if 1,0122

rhs = ones(N,1);

u = -sqrt(t)*log(1-(t*D+eye(N))\rhs);

Laplacian

I. PDE: Hopf-Cole transformation

20t 2t 2.0t

I. Applications: Dynamic distance-based shape features for gait recognition

T. P.Whytock, A. Belyaev, and N. M. Robertson, ”Dynamic distance-based shape features for gait recognition.” Journal of Mathematical Imaging and Vision. 2014.

I. Dynamic distance-based shape features for gait

recognition

I. Dynamic distance-based shape features for gait

recognition

II. Perona-Malik Diffusion

II. Perona-Malik diffusion with Matlab

P. Perona, T. Shiota, and J. Malik, “Anistropic Diffusuion.” Geometry-Driven Diffusion in Computer Vision, 1994.

KIIgIIgtI exp,div

II. Repeated averaging and nonlinear diffusion

Gray-scale image

Iterative local averaging:

Gaussian smoothing edge-enhancing averaging

),( yxIz

ij

ij

wkjyixIw

kyxI ),,(1

)1,,(

)|),,(|exp(),,( 2kyxIckyxwij 1),,( kyxwij

II. Repeated averaging and nonlinear diffusion

Gray-scale image

Iterative local edge-enhancing averaging:

),( yxIz

ij

ij

wkjyixIw

kyxI ),,(1

)1,,(

)),,(exp(),,( KkyxIkyxwij

Perona-Malik diffusion:

KIIg

IIgtI

exp

div Efficient numerical schemes Possibilities for various generalizations and improvements

II. Perona-Malik diffusion and its extensions

nonlinear diffusion

can be used for enhancing small-scale details

II. Nonlinear diffusion for mesh processing

2D Image Triangle mesh

),( yxI )(Tn

II. Nonlinear diffusion for surface denoising

Smoothing normals

Updating vertex positions

)(

)(

ineij

j

ineij

oldjj

newi w

w n

n

2

expT

kw j

j

0 jk

jw

Y. Ohtake, A. Belyaev, and I. A. Bogaevski, “Mesh Regularization and Adaptive Smoothing.” Computer-Aided Design, Vol. 33, No. 11, 2001, pp. 789–800.

II. Perona-Malik nonlinear diffusion for surface denoising

Nonlinear diffusion of mesh normals

Gaussian like smoothing

Adding noise

II. Perona-Malik nonlinear diffusion for surface denoising

Nonlinear diffusion of mesh normals

Conventional mesh smoothing

II. Perona-Malik nonlinear diffusion for surface denoising

II. Image compression with nonlinear diffusion

I. Galić, J. Weickert, M. Welk, M. Bruhn, A. Belyaev, H.-P. Seidel: “ “Image compression with anisotropic diffusion”. Journal of Mathematical Imaging and Vision. . 31(2-3): 255-269, 2008.

)(0,,1

11,,gdiv: 222

xfxufuxcLuxcu

ssgGuuuuuLu

t

T

III. Intro to Variational Image Processing: gradient

2

01

11

sup

,,

,,,,

h

hff

htxfdt

dhf

x

f

x

ff

xxxxxf

tn

nn

III. Intro to Variational Image Processing: max / min

function a is wheremax/min uuE

hhxxx

xx

any for 0 if at extremuman has

,,

000

1

t

d

tfdt

df

xxf

vtvuEdt

d

t

any for 00

III. Membrane energy

00

0,22

1

2

1

2

2

0

2

2

222

2

dx

udtvuE

dt

d

vdx

dudx

dx

udvdx

dx

dv

dx

du

t

uEtvuE

bvavdxdx

dvtdx

dx

dv

dx

dutdx

dx

dutvuE

dxdx

duuE

t

bx

ax

b

a

b

a

b

a

b

a

b

a

b

a

Minimizing E(u) by gradient descent:

xIxux

u

t

u

0,2

2

const, ttxuSo we have to stop this gradient descent flow at some t=T

III. Membrane Energy

Minimizing Eλ(u) by gradient descent: uI

x

u

t

u

2

2

xutxu t

,

uIx

u2

2

0

00

0,

2

1

2

2

0

2

2

2

22

Iudx

udtvuE

dt

d

uvdxIudx

udvdxvIudx

dx

dv

dx

du

t

uEtvuE

bvavtdxvIudxdx

dv

dx

dutuEtvuE

dxIudxdx

duuE

t

bx

ax

b

a

b

a

b

a

b

a

b

a

b

a

b

a

02

2

Iudx

ud

III. Variational Approach to Image Smoothing

22

22

1

,~

,~

in ,,,

min,,,

Iu

yxIyxuyxu

dxdyyxuyxuyxI

s

ssgdu

uugdivt

u

u

,min

Links to robust statistics

III. Variational Approach to Image Smoothing

yu

xuu

Resembles least-square fitting

Given image I(x,y), we approximate it by u(x,y)

min,,,22

dxdyyxuyxuyxIuE

data fitting term smoothing term

Energy (functional)

We have to learn how to differentiate E(u) w.r.t. u(x)

λ controls the amount of smoothing we add to I(x,y)

III. Edge-preserving image smoothing

min,,,2

dxdyyxuyxuyxIp

ppdxdyyxu

u

1~,

1~

preserved are edges image1

smoothed are edges image1

p

p

III. Total Variation Energy

ynumericall 0 Solving

222

,,,

2322

22

2222

uE

uu

uuuuuuuIu

u

uIuE

uuIuuuuLdxuIuuE

TV

yx

xyyxyyxyxx

yxyxTV

pppppkkkdxu

u

250250,505

50,50,50,50,50

1

ppdxu

u

250

0,0,250,0,0

edges sharp prefers 1 blur, prefers 1 pp

III. The Rudin-Osher-Fatemi (ROF) model

min,,, 2

dxdyyxuyxuyxI

02

div

Iuu

u

III. The Rudin-Osher-Fatemi (ROF) model

B.Goldlücke, Foundations of Variational Image Analysis, Lecture Notes, 2011

III. The Rudin-Osher-Fatemi (ROF) model

III. The Rudin-Osher-Fatemi (ROF) model

02

div,,

2div

2div,,,,

,,,,,,

2div,,

1

Iuu

uyxuu

Iuu

uuu

Iuu

utyxutyxu

tyxutyxutyxu

t

Iuu

utyxu

t

kk

kk

kkk

III. TV image processing models

(TV) Variation Total

22

dxdyyuxudu xx

III. Gradient descent minimization

min,

dxdyyxu

u

uuE

t

udiv uuE

t

u

min,2

dxdyyxu

Curvature flowLinear diffusion

III. The Rudin-Osher-Fatemi (ROF) model

Diffusion (heat) Total variation

Original signal

III. TV image inpainting

B.Goldlücke, Foundations of Variational Image Analysis, Lecture Notes, 2011

Original image I(x,y) Removed region R Inpainted result

min\

2

xxxx duduIR

IV. Image Deblurring

Image restoration is to restore a degraded image back to the original image

Linear image degradation model

( , ) ( , ) ( , ) ( , )g x y f x y h x y n x y blur additive noise

IV. A variational approach to image deblurring

GH

HF

FHGHF

dFdGFH

fgAf

gAffhAf

2

*

*

22

22

equation LagrangeEuler0

min

min

,

Wiener filtering

IV. Variational image deblurring

ω

ωω

ωω

ωωωωωωω

ωωω

GbaH

HF

dFbadGFH

dFbafL

fLgAfgAffhAf

2

*

22

2

2

min

min,,

IV. TV deblurring

iterationspoint fixed

0

equation LagrangeEuler

0

min,

12

1

2

22

gfhhf

f

gfhhf

f

dxffLfLgfh

n

n

n

non-blind deblurring

blind deblurring

IV. TV deblurring

V. Snakes: Active Contour Models

V. Geodesic active contours

V. Geodesic active contours

22, dydxyxgdl

Riemannian metric (conformal, for the sake of simplicity)

),(1

1),(

yxIyxg

V. Geodesic active contours

V. Geodesics in heat

Possibly this approach can be used for a very efficient implementation of geodesic active contours.

VI. B.K.P.Horn: Shape from Shading

Berthold Klaus Paul Horn, Robot Vision. The MIT Press. 1986.

VI. B.K.P.Horn: Shape from Shading

VII. Mumford-Shah Approach

VII. Blake-Zisserman = Mumford-Shah

VII. Chan-Vese active contours without contours

CoutsideCinside

dxdycyxudxdycyxuCFCF2

20

2

1021 ,,

The end.Thank you!