Robust Kernel Estimation for Single Image Blind Deconvolution

Fang Wang1,2 Yi Li2

fang.wang@nicta.com.au yi.li@nicta.com.au1 Nanjing University of Science and Technology

2 National ICT Australia (NICTA)

Abstract

Most of the state-of-the-art algorithms of restoringsingle blurred image are sensitive to image noise andartifacts. Our idea is to learn an adaptive filter forblind deconvolution to remedy this problem. We use thisauxiliary filter to progressively suppress image noise inearly stage of kernel estimation, leading to a robust ker-nel estimation algorithm. Our approach can naturallyhandle image noise and improve performance of sin-gle image blind deconvolution. Experiments suggestthat our approach is more robust than other methodsin restoration of blurred images.

1. Introduction

In consumer level cameras, motion blur caused bycamera shake is often unavoidable in low illuminationphotography conditions. Automatically restoring theseblurred photos to clean and artifact-free images is at-tractive and practically useful, and this becomes an op-eration called image deblurring or image deconvolution.

Most of state of the art algorithms focus on highquality images. However, large amount of blurred im-ages are in low quality due to various degradations, suchas noise and compression artifacts. Current availabledeconvolution algorithms cannot properly handle imagenoise. Estimation of blur kernels and restoring such lowquality images can be challenging.

Image noise is the main cause of error in kernel es-timation and artifacts in recovered images. Fig.1 showsthat noise leads to kernel estimation errors and pro-duces deblurring artifacts in result image. We gener-ated a synthetic noiseless blurred image (Fig.1b) usingreal blur kernel, and then compressed the blurred imageusing JPEG (Quality=80) to obtain a noisy correspon-dence. Fig.1c and 1d show blind deconvolution resultsof noiseless and noisy image, respectively, using a re-cently proposed method [5]. Artifacts are obvious in

(a) (b) (c) (d) (e)

Figure 1. Robust blind deconvolution. (a)Original image. (b) Synthetic blurred in-put. (c)&(d) Deconvolution result of noise-less and noisy images without robust en-hancement. (e) Our result on noisy image.

the result of noisy image. Thus, finding a robust kernelestimation method is necessary.

Linear filtering is a common approach in image de-noising, especially when no knowledge is given aboutthe noise. Our approach is to suppress noise by smooth-ing the input image with a learned linear filter. Insteadof using manually designed pre-processing filters, weproposed a method that automatically learns this auxil-iary filter, simultaneously with kernel estimation proce-dure. The auxiliary filter is adaptive to input images. Itis able to automatically handle image noise and lead toa robust kernel estimation algorithm. Fig.1e shows ef-fectiveness of our approach in deblurring noisy image.

2. Related work

Blind image deconvolution is a challenging ill-posedproblem. Errors in kernel estimation result can causesignificant ringing artifacts in restored image. Manyprevious works on blind deconvolution primarily focuson image priors. Fergus et al. [3] used mixture of Gaus-sian model as global probabilistic constraint of edge dis-tribution. Krishnan et al. [5] proposed a normalizedsparsity measurement which favors nature clear imagemore than blurred ones.

21st International Conference on Pattern Recognition (ICPR 2012)November 11-15, 2012. Tsukuba, Japan

978-4-9906441-1-6 ©2012 IAPR 481

Due to the highly ill-posed nature of the problem,standard maximum a posteriori based methods do notwork well. Numbers of complex techniques has beenproposed to enhance performance of blind deconvolu-tion. Cho et al. [1] used shock filter to gain estimationof clear images. Xu et al. [10] used a selective edgemap to help kernel estimation.

The performance of image deconvolution is highlysensitive to noise, which is the main cause of artifacts inrecovered image for both blind and non-blind deconvo-lution methods. Several methods have been proposed tohandle noise in deconvolution. Shan et al. [8] modeledimage noise by minimizing errors caused by inaccurateblur kernel estimation. Cho et al. [2] explicitly mod-eled outliers such as saturated pixels and non-Gaussiannoise. They classified outlier pixels using expectation-maximization algorithm and excluded these pixels indeconvolution to prevent artifacts.

3. Robust blind deconvolution approach

Blurred image g can be modeled as a convolution ofthe true clear image a and a blur kernel h plus noise n0:

g = a ∗ h+ n0, (1)

where (∗) is convolution operator. Blind deconvolutiontechniques aim at solving a from its blurred correspon-dence g without knowledge of h.

To suppress the noise term in Eq.1, we solve blinddeconvolution problem by smoothing the input imagewith a learned filter. Our approach is to convolve bothside of Eq.1 with an auxiliary filter f , then the equationcan be expressed as:

g ∗ f = a ∗ h ∗ f + n0 ∗ f= a ∗ k + n

(2)

where f is the auxiliary filter, k = h∗f is the smoothedkernel. For simplicity we use k as the blur kernel di-rectly. Assuming noise term n is gaussian will lead to aleast-square-error minimization approach to solve Eq.2by minimizing ‖g ∗ f − a ∗ k‖2. We will show howto perform this optimization in Sec.3.1. Our algorithmlearns filter f simultaneously with the blur kernel k, andrecovers the latent clean image a.

Our approach is particularly useful in handling noisein image gradients. Many deconvolution methods solvethe problem in image gradients [1, 5, 10], which makesthem very sensitive to noise. Our solution naturally han-dles this issue. The adaptive filter automatically sup-presses noise in these methods and improves their per-formances. In this paper, we choose [5] as our solver.

3.1. Kernel estimation

Our aim is to solve blind deconvolution problem de-fined in Eq.2. We firstly perform kernel estimationon image high frequencies using our newly proposedmodel. This can be solved using a robust optimizationapproach. Clear image is restored using a non-blind de-convolution module.

We firstly perform kernel estimation on imagederivatives. This is performed by minimizing the fol-lowing energy function:

E(k) = ‖∂∗a ∗ k − ∂∗g ∗ f‖22 + λ1‖∂∗a‖1‖∂∗a‖2

+ λ2‖k‖1 + λ3‖f‖1.(3)

where ‖ · ‖1, ‖ · ‖2 denotes l1 and l2 norms, λ1, λ2, λ3are weights that control image and kernel regularizationterms, and ∂∗ = [∂x, ∂y] denotes image gradients alongx and y directions. Here, we use high pass filters [−1, 1]and [−1, 1]T to compute image gradients.

Our model of the deconvolution process (Eq.3) isnon-convex when a, k, and f are unknowns. There-fore, iterative methods are frequently adopted to solvethese variables in an alternative manner. This amountsto solve the following three optimizations alternately:

1. Given k, f and g, solve a as follows:

argmina‖∂∗a ∗ k − ∂∗g ∗ f‖22 + λ1

‖∂∗a‖1‖∂∗a‖2

, (4)

2. Given a, f and g, solve k as follows:

argmink‖∂∗a ∗ k − ∂∗g ∗ f‖22 + λ2‖k‖1, (5)

3. Given a, k and g, solve f as follows:

argminf‖∂∗a ∗ k − ∂∗g ∗ f‖22 + λ3‖f‖1. (6)

Though solving Eq.3 involves a complicate non-convex optimization with large number of variables.The sub-optimization problems Eq.5 and Eq.6 are con-vex. Although the sub-optimization problem in Eq.4 isnon-convex due to the normalized sparse prior, it canbe transformed to a convex l1 constrain formulationby fixing the l2 norm denominator from previous iter-ation. Please refer to [5] for details. We also add phys-ical validate constrains on both k and f , which includek ≥ 0,

∑i ki = 1; f ≥ 0,

∑i fi = 1.

Here we describe how the additional filter f can fa-cilitate kernel estimation procedure. As we mentionedbefore, the performance of kernel estimation is subjectto noise in images. The smoothing filter f can reduce

482

the effect of noise and estimation errors. To achieve abetter estimation of k in the first several iterations, weinitialize filter f as an isotropic Gaussian filter with thesame size as k. Then during the estimation procedure,we gradually shrink down size of f and increase theweight of sparsity regularization term of f . A largerweight of sparsity term makes the filter f favors a deltafunction and leads to a more precise result of k.

3.2. Image restoration

Once we have the blur kernel estimation, we canrestore the blurred image using any non-blind decon-volution methods. In our implementation, we used arecently proposed deconvolution method [4]. This al-gorithm uses a lookup table technique for computationacceleration, which is much more effective than otherdeconvolution algorithms while producing reasonablegood results in image deblurring.

3.3. Multi-scale approach

Most of blind deconvolution methods need to be de-signed properly to avoid degenerate solutions such asa = g; k = δ. Hence, we performed kernel estima-tion using a coarse-to-fine method similar to other blinddeconvolution methods [2, 3, 10].

7×7 11 17 25 37 53 55

Figure 2. Multi-scale kernel estimation.Without linear filtering (top) and with lin-ear filtering (bottom). Kernel sizes of dif-ferent scales are shown below. Imagesare color coded, best viewed in color.

We build multi-scale pyramids for both images andkernels (both k and f ). For the coarsest level, we setkernel size as 3 × 3, then each level can be generatedby up-sampling the previous level by factor

√2. Fig.2

illustrate the auxiliary filter improved kernel estimationresults in first several stages. From coarse to fine (leftto right in Fig.2), our method progressively suppressesnoise and weak image edges that are less helpful tokernel estimation, while still maintaining main imagestructures and strong edges. This produced a stable andfast kernel estimation results in coarse levels. Our strat-egy results in a significant difference in image restora-tion (Fig.5d and 5e, respectively).

(a) Clear (c) Levins (e) Cho

(b) Blurred (d) Ours (f) Krishnan

Figure 3. Example results of comparison.

4. Experiments

4.1. Robustness evaluation

To demonstrate our approach outperforming state-of-the-art methods in robustness, we evaluated our re-sults on Levin et al.’s blur image dataset [6] againstthree other methods. The dataset has 4 different imageswith size 255 × 255, and each blurred with 8 differentkernels, which makes 32 images in total. Fig.3a and3b show example images of the dataset. We used bothpixel wise errors and Universal Quality Index (UQI) [9]for robustness evaluation. We first computed pixel wisedifference of each restored image against ground truth,and averaged errors over each image. For UQI mea-surements, we used the authors’ matlab implementa-tion. The accumulative percentages of image with re-gard to average pixel errors and “1-Quality” are shownin Fig.4. Higher percentage in smaller avg. error rangeor larger UQI quality range means algorithm producedbetter results on more images, which reveals the robust-ness of our approach. See sample results in Fig.3c-3f.

0.02 0.035 0.050

25

50

75

100

Accum

ula

tive p

erc

enta

ges (

%)

Avg. Errors

Average Error

Our approach

Krishnan

Cho

Levin

0.12 0.21 0.30

25

50

75

100

Accum

ula

tive p

erc

enta

ges (

%)

1−Quality

Universal Quality Index

Our approach

Krishnan

Cho

Levin

Figure 4. Error ratios comparison of fourmethods on Levin’s dataset.

483

(a) Input (b) Levin (c) Cho (d) Krishnan (e) Our method (f)

Figure 5. Result Comparison. (a) blurred images. (b)-(d) deblurring results of Levin et al.[7],Cho et al.[1], and Krishnan et al.[5], respectively. (e) our restoration results. Close-up patchesof comparison between original inputs (top) and our results (bottom) is shown in column (f).

4.2. Image deconvolution results

We then performed blind image deconvolution onimages with noise and compression artifacts. The testimages were captured using hand-held DSLR with 0.3sshutter speed. To get noisy inputs, all test images are re-sized to 800× 533 pixels and JPEG compressed (Qual-ity=80). We compared our deblurring results with threestate-of-the-art methods including Levin et al. [7], Choet al. [1], and Krishnan et al. [5]. Fig.5 shows therestored results and estimated kernels of all four meth-ods. Close-up patches highlighted our results with orig-inal inputs. Notice the highlight in bird’s eye revealedtrue blur kernel shape, which shows that our methodcorrectly estimated blur kernels and recovered details.Benefited from the auxiliary filter, our algorithm pro-duced robust kernel estimation results, which were sim-ilar with the true blur kernels.

5. Conclusion

We proposed an improved kernel estimation methodfor single image blind deconvolution. By adding anadaptive linear filter, our method progressively sup-pressed noise in early stage of kernel estimation, andachieved more robust estimation results. Recovered im-ages using our method are clean and with less deblur-ring artifacts. Experiment results showed our algorithmis more robust than state-of-the-art algorithms, and hasbetter performance in blind image deconvolution.

References

[1] S. Cho and S. Lee. Fast motion deblurring. In Transac-tions on Graphics, volume 28, page 145. ACM, 2009.

[2] S. Cho, J. Wang, and S. Lee. Handling outliers in non-blind image deconvolution. In ICCV, 2011 IEEE Con-ference on, pages 495–502. IEEE, 2011.

[3] R. Fergus, B. Singh, A. Hertzmann, S. Roweis, andW. Freeman. Removing camera shake from a singlephotograph. In Transactions on Graphics, volume 25,pages 787–794. ACM, 2006.

[4] D. Krishnan and R. Fergus. Fast image deconvolutionusing hyper-laplacian priors. Advances in Neural Infor-mation Processing Systems, 22:1–9, 2009.

[5] D. Krishnan, T. Tay, and R. Fergus. Blind deconvolutionusing a normalized sparsity measure. In CVPR, 2011IEEE Conference on, pages 233–240. IEEE, 2011.

[6] A. Levin, Y. Weiss, F. Durand, and W. Freeman. Under-standing and evaluating blind deconvolution algorithms.In CVPR 2009. IEEE Conference on, pages 1964–1971.Ieee, 2009.

[7] A. Levin, Y. Weiss, F. Durand, and W. Freeman. Effi-cient marginal likelihood optimization in blind decon-volution. In CVPR, 2011 IEEE Conference on, pages2657–2664. IEEE, 2011.

[8] Q. Shan, J. Jia, and A. Agarwala. High-quality mo-tion deblurring from a single image. In Transactions onGraphics, volume 27, page 73. ACM, 2008.

[9] Z. Wang and A. Bovik. A universal image quality index.Signal Processing Letters, IEEE, 9(3):81–84, 2002.

[10] L. Xu and J. Jia. Two-phase kernel estimation for robustmotion deblurring. ECCV 2010, pages 157–170, 2010.

484