blur as a chance and not a nuisancecmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/... ·...
TRANSCRIPT
[email protected] www.utia.cas.cz
Institute of Information Theory and Automation
Academy of Sciences of the Czech Republic Prague
Digital Image Restoration Blur as a chance and not a nuisance
Filip Šroubek
1 CMP 2017
2 CMP 2017
original image
u
Image Degradation Model
3
channel
convolution
acquired image
= z + n
noise
CMP 2017
Energy Minimization
CMP 2017 4
Data term
Energy Minimization
5 CMP 2017
Data term
Blur regularization
• Blur has different shapes • Compact support • Non-negative • Preserve energy
Energy Minimization
6 CMP 2017
Motion blur
Out-of-focus & Abberations
Image regularization
Energy Minimization
7 CMP 2017
Data term
0 0.2 0.4 0.6 0.8 1
-0.5 0 0.5
Statistics of sharp images
CMP 2017 8
Intensity
Gradient
-0.5 0 0.5
0 0.2 0.4 0.6 0.8 1
Statistics of blurred images
CMP 2017 9
Intensity
Gradient
Regularization
CMP 2017 10
• “NO-BLUR” solution: • WHY? • Regularization favors blurred images
Energy Minimization
11 CMP 2017
Regularization favors blur
CMP 2017 12
5 10 15 200.5
1
1.5
Regularization favors blur
CMP 2017 13
5 10 15 200.5
1
1.5
Artificially sparsify images
Ad-hoc steps!
• To avoid “no-blur” solution:
• Artificial sharpening • Remove spikes • Adjusting priors on the fly • Hierarchical approach
CMP 2017 14
Chan TIP98 Shan SigGraph08 Cho SigGraph09 Xu ECCV09, CVPR13 Almeida TIP10 Krishnan CVPR11 Zhong CVPR13 Sun ICCP13 Michaeli ECCV14 Perrone PAMI15 Pan CVPR16
Artificial Sharpening
15
Shock filter Blur prediction
h-step
Deconvolution u-step
Blurred image
Cho et al., SIGGRAPH 2009
CMP 2017
Remove Spiky Objects
16
Xu et al., ECCV 2010
CMP 2017
Remove Spiky Objects
CMP 2017 17
Mask out small objects
Xu et al., ECCV 2010
Remove Spiky Objects
CMP 2017 18
Reconstructed image with small objects removed
Xu et al., ECCV 2010
Hierarchical Deconvolution
19
Scale N-1
Scale N-2
Scale N
image blur
CMP 2017
Bayesian Paradigm
a posteriori distribution unknown
likelihood given by our problem
a priori distribution our prior knowledge
20 CMP 2017
Blind deconvolution with MAP
21 CMP 2017
Bayesian Paradigm revisited
• Marginalize the posterior
• Maximize the marginalized prob.
• Then maximize the posterior
CMP 2017 22
CMP 2017 23
-0.4-0.6
-0.8-1
-1.21
0.5
0
-0.5
-1
0.5
0.6
0.7
0.8
0.9
1
no-blur solution
correct solution
How to marginalize?
• If Gaussian distributions analytic solution exists in the form of Gaussian distribution
• If not (our case) approximation • Laplace approximation • Factorization with Variational Bayes
CMP 2017 24
Variational Bayes
• Factorization of the posterior
and then marginalization is trivial. • Every factor q depends on moments of other
variables => must be solved iteratively.
CMP 2017 25
Miskin AICA00 Fergus SIGGRAPH06 Tzikas TIP09 Whyte IJCV12 Levin PAMI11 Babacan ECCV12 Wipf JMLR14
Marginalization in blind deconvolution
26 CMP 2017
Limitations of BD
• Gaussian noise
CMP 2017 27
original
SNR=20dB SNR=50dB
Limitations of BD
• Model discrepancies
CMP 2017 28
original
7% discrepancy
Automatic Relevance Determination (Students’ t-distribution) • Standard data term
replaced by
CMP 2017 29
CMP 2017 30
Space-variant Blur
CMP 2017 31
Camera motion
Optical aberrations
Object motion
Scene depth
Approximation of SV Blur
• Patch-wise convolution • General SV PSF • Locally convolution may not hold
• Parametric model (Blur Basis) • More accurate • Model may not hold
• Conversion to space-invariant • HW design
• Local object motion • Segmentation • Line blur
CMP 2017 32
Joshi CVPR08, Sorel ICIP09, Ji CVPR12,Sun CVPR15
Levin NIPS06, Shan ICCV07, Dai CVPR08, Chakrabarti CVPR10, Kim CVPR14
Whyte CVPR10, Gupta ECCV10, Hirsch ICCV11, Zhang NIPS13
Levin SIGGRAPH08, Ben-Ezra PAMI04, Tai PAMI10, Wavefront coding
Fast Moving Objects
• FMO moves over a distance exceeding its size within exposure time
CMP 2017 33
D.R.,J.K.,F.S.,L.N.,J.M. arXiv:1611.07889
Formation model
CMP 2017 34
Alpha matting
FMO Appearance Estimation
CMP 2017 35
FMO Localization
• The pipeline consists of three stages:
CMP 2017 36
1) Explorer 2) Detector 3) Tracker
Input FMO Video
CMP 2017 37
FMO Tracking + Reconstruction
CMP 2017 38
Temporal SR - Table tennis
CMP 2017 39
Temporal SR - Darts
CMP 2017 40
Rotating FMOs
• Simple convolution
replaced by space-variant convolution is a function of trajectory and angular velocity is a 3D object (sphere) • Gradient descent w.r.t • Exhaustive search w.r.t angular velocity
CMP 2017 41
Temporal SR with Rotation
42 CMP 2017
Input video frame Estimated high-speed video
Limitations • Poor collaboration between tracking and restoration
• Estimated blur and FMO appearance improve tracking
• Unstable FMO appearance estimation • Combine multiple video frames
• Track multiple FMOs
CMP 2017 43
Thank You
For Your Attention
44 CMP 2017
original image
u
acquired images
= zk
channel K
channel 2
channel 1
[u * hk]
Multichannel Model
+ nk
noise
46 CMP 2017
Image regularization
term
Blur Regularization
term
Data term
Gaussian noise
L2 norm
Multichannel Model
• Acquisition model:
• Optimization problem
47 CMP 2017
0
Multichannel Blur Regularization
u
h2 * u u h1 * = = z1 z2
48
z1 * * u h1 = * z2 * h2 u * = *
CMP 2017
Camera motion
49 CMP 2017
Degraded images
Reconstructed image
Astronomical images
PSF estimation
50 CMP 2017
original image
u
acquired images
= zk
channel K
channel 2
channel 1
[u * hk]
MC Model with Decimation
+ nk
noise
D
51 CMP 2017
Robustness to misalignment
52
original image PSF degraded image
CMP 2017
Super-resolution
53
Superresolved image (2x) Optical zoom (ground truth)
rough registration
CMP 2017
8 images
SR 3x
SR 2x
original
CMP 2017 54
Input video Super-resolution
Video Super-resolution
55 CMP 2017
Embedded Systems
56 CMP 2017
Acquired blurred image
57 CMP 2017
Blur estimation
58 CMP 2017
Patch-wise Deconvolution
59 CMP 2017
Super-resolution
CMP 2017 60
JPEG SR
Embedded Super-Resolution
Input image
SR from 5 images
SR from 10 images
61 CMP 2017
Patch-wise restoration
62 CMP 2017
63 CMP 2017
Regularization
CMP 2017 64
Adjusting priors
65
• Start with an overestimated noise level and slowly decrease it to the correct level.
• Start with p<<1 and slowly increase it to p=1. CMP 2017