Rob Fergus

Courant Institute of Mathematical Sciences

New York University

A Variational Approach to Blind Image Deconvolution

Overview

• Much recent interest in blind deconvolution:

• Levin ’06, Fergus et al. ’06, Jia et al. ’07, Joshi et al. ’08, Shan et al. ’08, Whyte et al. ’10

• This talk:

– Discuss Fergus ‘06 algorithm

– Try and give some insight into the variational methods used

Removing Camera Shake from a Single Photograph

Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T. Roweis and William T. Freeman

Massachusetts Institute of Technology and

University of Toronto

Image formation process

=

Blurry image Sharp image

Blur kernel

Input to algorithm Desired output

Convolutionoperator

Model is approximationAssume static scene & constant

blur

Why such a hard problem?

=

Blurry image

Sharp image Blur kernel

=

=

Statistics of gradients in natural images

Histogram of image gradients

Characteristic distribution with heavy tails

Log

# pi

xels

Image gradient is difference between spatially adjacent pixel intensities

Blurry images have different statistics

Histogram of image gradients

Log

# pi

xels

Parametric distribution

Histogram of image gradients

Use parametric model of sharp image statistics

Log

# pi

xels

Three sources of information

1. Reconstruction constraint:

=

Input blurry imageEstimated sharp imageEstimatedblur kernel

2. Image prior: 3. Blur prior:

Positive&

SparseDistribution of gradients

Three sources of information

y = observed image b = blur kernel x = sharp image

Posterior

p(b;xjy) = k p(yjb;x) p(x) p(b)

Three sources of information

y = observed image b = blur kernel x = sharp image

Posterior 1. Likelihood(Reconstruct

ion constraint)

2. Image prior

3. Blurprior

p(b;xjy) = k p(yjb;x) p(x) p(b)

Three sources of information

y = observed image b = blur kernel x = sharp image

y = observed image b = blur x = sharp image

1. Likelihood Reconstruction constraint

p(yjb;x) =Q

i N (yi jxi b;¾2)i - pixel

index

Overview of model

2. Image prior p(x)

Mixture of Gaussians fit to empirical distribution of image gradients

3. Blur prior p(b)

Exponential distribution to

keep kernel +ve & sparse

The obvious thing to do

– Combine 3 terms into an objective function

– Run conjugate gradient descent

– This is Maximum a-Posteriori (MAP)

No success!

Posterior 1. Likelihood(Reconstruct

ion constraint)

2. Image prior

3. Blurprior

p(b;xjy) = k p(yjb;x) p(x) p(b)

Variational Independent Component Analysis

• Binary images

• Priors on intensities

• Small, synthetic blurs

• Not applicable to natural images

Miskin and Mackay, 2000

Variational Bayes

Variational Bayesian approach

Keeps track of uncertainty in estimates of image and blur by using a distribution instead of a single estimate

Toy illustration: Optimization surface for a single variable

Maximum a-Posteriori (MAP)

Pixel intensity

Score

Simple 1-D blind deconvolution example

y = observed image b = blurx = sharp image

n = noise ~ N(0,σ2)

Let y = 2

y = bx + n

Let y = 2 σ2 = 0.1

N (yjbx;¾2)

p(b;xjy) = k p(yjb;x) p(x) p(b)

Gaussian distribution:

N (xj0;2)

p(b;xjy) = k p(yjb;x) p(x) p(b)

p(b;xjy) = k p(yjb;x) p(x) p(b)

Marginal distribution p(b|y)

p(bjy) =R

p(b;xjy) dx = kR

p(yjb;x) p(x) dx

0 1 2 3 4 5 6 7 8 9 100

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

b

Bayes

p(b

|y)

0 1 2 3 4 5 6 7 8 9 100

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

b

Bayes

p(b

|y)

MAP solution

Highest point on surface:argmaxb;x p(x;bjy)

Variational Bayes

• True Bayesian approach not tractable

• Approximate posterior with simple distribution

Fitting posterior with a Gaussian

• Approximating distribution is Gaussian

• MinimizeK L (q(x;b) jj p(x;bjy))

q(x;b)

KL-Distance vs Gaussian width

0 0.1 0.2 0.3 0.4 0.5 0.6 0.74

5

6

7

8

9

10

11

Gaussian width

KL(

q||

p)

Variational Approximation of Marginal

0 1 2 3 4 5 6 7 8 9 100

0.5

1

1.5

2

2.5

b

p(b

|y)

Variational

True marginal

MAP

Try sampling from the model

Let true b = 2

Repeat:

• Sample x ~ N(0,2)

• Sample n ~ N(0,σ2)

• y = xb + n

• Compute pMAP(b|y), pBayes(b|y) & pVariational(b|y)

• Multiply with existing density estimates (assume iid)

0 1 2 3 4 5 6 7 8 9 100

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

b

p(b

|y)

Actual Setup of Variational Approach

Work in gradient domain:

! r x b= r yx b= y

Approximate posterior with

is Gaussian on each pixel

is rectified Gaussian on each blur kernel element

p(r x;bjr y)q(r x;b)

q(r x;b) = q(r x)q(b)q(r x)q(b)

K L (q(r x)q(b) jj p(r x;bjr y))

Assume

Cost function

Close-up

• Original

• Output

Original photograph

Our output

Blur kernel

Blur kernelOur outputOriginal image

Close-up

Recent Comparison paper

Percent success

(higher is better)

IEEE CVPR 2009 conference

Related Problems

• Bayesian Color Constancy

– Brainard & Freeman

• JOSA 1997

– Given color pixels, deduce:

• Reflectance Spectrum of surface

• Illuminant Spectrum

– Use Laplace approximation

– Similar to Gaussian q(.) From Foundation of Vision by Brian Wandell, Sinauer

Associates, 1995

Conclusions

• Variational methods seem to do the right thing for blind deconvolution.

• Interesting from inference point of view: rare case where Bayesian methods are needed

• Can potentially be applied to other ill-posed problems in image processing