high-quality motion deblurring from a single image...

High-quality Motion Deblurringfrom a Single Image

High-quality Motion Deblurringfrom a Single Image

QiQi ShanShanLeo Leo JiayaJiaya JiaJiaAseemAseem AgarwalaAgarwala

The Chinese University of Hong KongThe Chinese University of Hong KongAdobe Systems, Inc.

2

The Problem

3

Blur Model

Blurred image I Sharp image L

Camera Noise n

Blur kernel f

4

Previous Work (1)

Hardware solutions:

[Raskar et al. 2006]

[Ben-Ezra and Nayar 2004]

[Levin et al. 2008]

5

Previous Work (2)

Multi-frame solutions:

[Petschnigg et al. 2004]

[Jia et al. 2004] [Rav-Acha and Peleg 2005]

[Yuan et al. 2007]

6

Previous Work (3)

Single image solutions:

[Jia 2007][Fergus et al. 2006]

[Levin et al. 2007] [Yuan et al. 2008]

7

The Problems

Non-blind deconvolution

Blind deconvolution

An Example

9

Challenges (1)

Assuming no noise

10

Challenges (2)

With noise

An Example

An Example

An Example

An Example

15

n

16

n

17

Noise constraint

n

18

0( | 0, )iiN n ζ∏

1( | 0, )iiN n ζ∇∏

0 1( | 0, ) ( | 0, )i ii iN n N nζ ζ∇∏ ∏

Noise constraint

n

Possible noise models:

(1)

(2)

19

0 1

2

( | 0, ) ( | 0, )

( | 0, )i ii i

ii

N n N n

N n

ζ ζ

ζ

∇

∇∇

∏ ∏∏

Noise constraint

n

0( | 0, )iiN n ζ∏

1( | 0, )iiN n ζ∇∏

Possible noise models:

(1)

(2)

20

0 1

2

( | 0, ) ( | 0, )

( ( )

(

| 0, )

) i ii i

ii

P n N n N n

N n

ζ ζ

ζ

= ∇

∇ ∇

∏ ∏∏

A random variable following an independent Gaussian distribution also has its any order derivative following it. [Simon 2002]

n

21

n

22

n

23

n

24

exponentially distributed

, ( 0) jfj

i

ep f fτ−= ≥∏

Kernel Statistics

f

25

L

26

L

Prior p(L) trabajando en dos escalas:Global: para ayudar a mitigar la

ill-possedness del problema (a nivel estadístico)

Local: para corregir los ringingartifacts

27

Image Global Statistics

L

…

28

Image Global Statistics

L

…

29

Image Global Statistics

…

L

30

Image Global Statistics

L

21

| |( ( )

( ))

k L x ca L

P Lb x c

− ∇ ≤⎧= ⎨− ∇ + >

∇⎩

))(log( 1 LP ∇

31

Image Global Statistics

L

21

| |( ( )

( ))

k L x ca L

P Lb x c

− ∇ ≤⎧= ⎨− ∇ + >

∇⎩

))(log( 1 LP ∇

32

Image Local Constraint

L

33

Image Local Constraint

L

34

LI

Image Local Constraint

L

35

LI

Image Local Constraint

L

36

LI

Image Local Constraint

L

37

LI

Image Local Constraint

12 ( | 0,) )(i white

p N L IL σ∈

= ∇ −∇∏

L

∏∈

∇−∇whitei

ii ILN ),0|( 1σ

38

LI

Image Local Constraint

12 ( | 0,) )(i white

p N L IL σ∈

= ∇ −∇∏

L

∏∈

∇−∇whitei

ii ILN ),0|( 1σ

39

Combining All constraints

1 2min ( , ) min log[ ( ) ( ) ( ) ( )]E L f p n p L p L p f= ∇

L f n

Two-step iterative optimization• Optimize L• Optimize f

40

1 2( ) log[ ( ) ( ) ( )]E L p n p L p L= ∇

**

* * 22( ) || ||E L w L f I

∇∇

⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑

Convolution is time-consuming

log ( )p n

1log ( )p L∇ 2log ( )p L1 1 1|| log ( ) ||p Lλ ∇ 2

2 2( || || )i ii white

L Iλ∈

+ ∇ −∇∑

Optimize L

41

Idea: separate convolution

**

* * 22( ) || ||E L w L f I

∇∇

⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑

log ( )p n

Optimize L

1log ( )p L∇ 2log ( )p L1 1 1|| log ( ) ||p Lλ ∇ 2

2 2( || || )i ii white

L Iλ∈

+ ∇ −∇∑

42

Idea: separate convolution

**

* * 22'( ) || ||E L w L f I

∇∇

⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑

log ( )p n

Optimize L

1log ( )p L∇ 2log ( )p L1 1 1|| log ( ) ||pλ Ψ 2

2 2( || || )i ii white

L Iλ∈

+ ∇ −∇∑

43

Idea: separate convolution

**

* * 22'( ) || ||E L w L f I

∇∇

⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑

log ( )p n

Optimize L

1log ( )p L∇ 2log ( )p L1 1 1|| log ( ) ||pλ Ψ 2

2 2( || || )ii white

i Iλ∈

+ −∇Ψ∑

44

**

* * 22'( ) || ||E L w L f I

∇∇

⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑

log ( )p n

Idea: separate convolution

22(|| || )Lγ+ Ψ −∇

Optimize L

1log ( )p L∇ 2log ( )p L1 1 1|| log ( ) ||pλ Ψ 2

2 2( || || )ii white

i Iλ∈

+ Ψ −∇∑

45

**

* * 22'( ) || ||E L w L f I

∇∇

⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑

1 1 1|| log ( ) ||pλ Ψ 22(|| || )Lγ+ Ψ −∇2

2 2( || || )i ii white

Iλ∈

+ Ψ −∇∑

Iteratively optimize L:Update LUpdate Ψ

**

* * 22|| ||w L f I

∇∇

⎛ ⎞= ∇ ⊗ −∇⎜ ⎟⎝ ⎠∑ 2

2(|| || )Lγ Ψ −∇+

46

**

* * 22'( ) || ||E L w L f I

∇∇

⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑ 2

2(|| || )Lγ+ Ψ −∇

Plancherel’s theorem implies:the sum of the square of a function equals the sum of the square of its Fourier transform

Iteratively optimize L:Update LUpdate Ψ

47

**

* * 22'( ) || ||E L w L f I

∇∇

⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑

Iteratively optimize L:Update LUpdate Ψ

22(|| || )Lγ+ Ψ −∇

We minimize E’(F(L)) instead, with a closed-formsolution.

**

* * 22'( ( )) || ( ) ( ) ( ) ||E F L w F L F f F I

∇∇

⎛ ⎞= ∇ − ∇ +⎜⎝

• ⎟⎠

∑22(|| ( ) ( ) || )F F Lγ Ψ − ∇

48

Iteratively optimize L:Update LUpdate Ψ

**

* * 22|| ||w L f I

∇∇

⎛ ⎞∇ ⊗ −∇ +⎜ ⎟

⎝ ⎠∑

1 1 1|| log ( ) ||pλ Ψ 22(|| || )Lγ+ Ψ −∇2

2 2( || || )i ii white

Iλ∈

+ Ψ −∇∑

'( )E Ψ =

49

Iteratively optimize L:Update LUpdate Ψ

1 1 1|| log ( ) ||pλ Ψ 22(|| || )Lγ+ Ψ −∇2

2 2( || || )i ii white

Iλ∈

+ Ψ −∇∑'( )E Ψ =

'i

iE Ψ=∑

each only contains a single variable Ψi'i

E Ψ

50

Iteratively optimize L:Update LUpdate Ψ

Iteration 0Iteration 2Iteration 1Iteration 4Iteration 3Iteration 7Iteration 5Iteration 6Iteration 8 (converge)

51

Time: about 30 seconds for an 800x600 image

Iteration 8 (converge)

52

A comparison

RL deconvolution

53

A comparison

Our deconvolution

54

Two-step iterative optimization• Optimize L• Optimize f

**

* * 22 1( ) || || || ||E f w L f I f

∇∇

⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑

1 2min ( , ) min log[ ( ) ( ) ( ) ( )]E L f p n p L p L p f= ∇

A form of L1-norm regularized problem and is solved using an interior point method

55Iteration 0

56Iteration 1

57Iteration 6

58Iteration 10

59

Iteration 1

60

Iteration 2

61

Iteration 4

62

Iteration 6

63

Iteration 8

64

Convergence

Time: about 350 seconds for an 800x600 image

65

Results

66

Results

67

Results

68

Results

69

70

71

More results

72

More results

73

More results

74

More results

75

More results

76

More results

77

More results

78

More results

79

Running Time• Non-blind deconvolution (an 800 x 600 image)

• Blind deconvolution– Approximately 10 minutes for an 800 x 600 image

Approximately 30 seconds in our Matlab implementation

Less than 10 seconds in our C++ implementation

80

Program Available

81

Conclusion

• A well-constrained single-image deblurring model

• Advanced optimization

• Improvement is possible for– Spatially-variant blur– Saturated images

General Blur Images

83

General Blur Images

85

Initialization

• User specified initialization

• Multi-Scale approach– Initialize the clear image in each scale by the one

recovered in a upper layer

86

Color channel

• For a blur images with a large PSF, we take all the three channels (RGB) into computation

• For a blur image with a small PSF, only the luminance channel are taken into computation