![Page 1: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/1.jpg)
Recent advances in deblurring
and image stabilization
Michal Šorel
Academy of Sciences of the Czech
Republic
![Page 2: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/2.jpg)
Camera shake stabilization
• Alternative to OIS (optical image
stabilization) systems
• Should work even for subject motion
![Page 3: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/3.jpg)
Remote sensing example
![Page 4: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/4.jpg)
Talk outline
• How to describe the blur? (convolution,
velocity field, PSF )
• Hardware-based stabilization
• Software deblurring– Multiple underexposed/noisy images
– Non-blind restoration
– Single blurred image (deconvolution)
– Multiple blurred images (deconvolution)
– One blurred and one underexposed image
– Multiple images blurred by sideways vibrations
![Page 5: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/5.jpg)
What is an image?
• Rectangular grid of pixels
• Image is a matrix M × N for greyscale
images
• Matrix M × N × 3 for color images
• Formulas shown for greyscale images
![Page 6: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/6.jpg)
Image as a function
• In formulas often a real function of two
variables R2→R+ , mostly 0..1
![Page 7: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/7.jpg)
Pinhole camera model
Pinhole camera
(Camera obscura)
Pinhole camera
model
![Page 8: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/8.jpg)
Focal length and sensor size
fish-eye lens
f down to 5mm
telephoto lens
(f > 100 mm)
normal lens
f ~ 50 mm
![Page 9: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/9.jpg)
What happens if camera moves?
• Sharp image – movement less than ½ pixel
• Influence of focal length, shutter speed,
sensor resolution (pixel density)
• Velocity field, PSF ~ blur kernel
![Page 10: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/10.jpg)
3D camera motion
• Rigid body – 6 degrees of freedom
• Natural coordinate system
• 2 vectors of camera velocity:
![Page 11: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/11.jpg)
Roll, Yaw, Pitch movements
Pan ... follow an object by a camera (often
refers to horizontal motion)
![Page 12: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/12.jpg)
Rotation down - demonstration
![Page 13: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/13.jpg)
Camera rotates downwards ↓
(pitch motion)Velocity field
![Page 14: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/14.jpg)
Velocity field
d - depth map
![Page 15: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/15.jpg)
Rotation about optical axis
(roll)
![Page 16: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/16.jpg)
General 3D rotation
![Page 17: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/17.jpg)
Stabilizer of 3D camera rotation
• For hand shake, camera rotation is mostly
dominant
• Blur is independent of scene depth (that is
why optical image stabilizers can work)
and changes gradually
![Page 18: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/18.jpg)
Translation
![Page 19: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/19.jpg)
Translation along optical axis
![Page 20: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/20.jpg)
Point-spread function - PSF
• Integration of velocity field → PSF
h(s,t; x2,y2)
h(s,t; x1,y1)
(x2,y2)
(x1,y1)
![Page 21: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/21.jpg)
• PSF h ... depends on position (x,y)
• Generalized convolution
• Convolution case – h is called convolution
kernel or convolution mask
Mathematical model of blurring
![Page 22: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/22.jpg)
PSF for camera shake
h(s,t; x2,y2)
h(s,t; x1,y1)
h(s,t; x3,y3)
(x1,y1)
(x2,y2)
(x3,y3)
![Page 23: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/23.jpg)
Blur description – summary (I)
• What we have learned
– What happens when a camera is moving
– 4 motion compoments
– Velocity field
– How PSF describes the blur and its relation
with velocity field
![Page 24: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/24.jpg)
Blur description – summary (II)
Motion component Dependence on
distance
Space-variant blur
YAW, PITCH (x,y-
axis rotation)
NO YES (a bit)
ROLL (z-axis rotation) NO YES (a lot)
X,Y-axis translation YES NO
Z-axis translation YES YES (a lot)
![Page 25: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/25.jpg)
Talk outline
• How to describe the blur? (convolution,
velocity field, PSF )
• Hardware-based stabilization
• Software deblurring– Multiple underexposed/noisy images
– Non-blind restoration
– Single blurred image (deconvolution)
– Multiple blurred images (deconvolution)
– One blurred and one underexposed image
– Multiple images blurred by sideways vibrations
![Page 26: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/26.jpg)
Hardware approaches to
suppress blur
• Boosting ISO (100, 200, 400, 800, 1600,
3200)
• External stabilization/gyro-stabilized
gimbals (two principles)
• Optical image stabilization (OIS) systems
![Page 27: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/27.jpg)
High ISO is not a solution
• ISO - 100, 200, 400, 800, 1600, 3200
• ISO 100 ISO 200
~ f-number/2, 2*t (1 EV or 1 stop)
• ISO 100 ISO 3200 ~ 32*t (5 stops)
Photon noise (Poisson) SNR ~ SNR0* t
SNR1600 = SNR100 / 16 (-12 dB)
SNR3200 = SNR100 / 32 (-15 dB)
![Page 28: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/28.jpg)
SNR
30 dB
20 dB
15 dB
5 dB
![Page 29: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/29.jpg)
Gyro-stabilized gimbals
Gyron FS
(Nettmann systems international)http://www.camerasystems.com/gyronfs.htm
![Page 30: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/30.jpg)
Gyro-stabilized gimbals
(airborn)
SUPER G (Nettman)
Panavision, IMAX cameras
5-axis Aerial Camera
System
91 kg
up to 220 km/h
TASE (Cloud cap tech. -
for UAVs), 13x17x11 cm
0.9 kg
0.05° pointing resolution
f=32mm ~ 500pixelshttp://www.cloudcaptech.com
![Page 31: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/31.jpg)
Helicopter – external demo
![Page 32: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/32.jpg)
Gimbal stabilization - demo
![Page 33: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/33.jpg)
Stabilizer precision/resolution
prec = 0.05°
60°60° ~ 60/0.05 = 1200 pixels
30° ~ 30/0.05 = 600 pixels
![Page 34: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/34.jpg)
Hardware-based image
stabilization
• Optical image stabilization
– Canon (IS - Image stabilization)
– Nikon (VR – Vibration Reduction)
– Panasonic, Leica, Sony, Sigma, Tamron,
Pentax ....
• Moving sensor
– Konika-Minolta (Sony –line)
– Olympus
![Page 35: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/35.jpg)
Image stabilization
www.canon.com
![Page 36: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/36.jpg)
Nikon VR
http://imaging.nikon.com/products/imaging/technology/vr/index.htm
![Page 37: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/37.jpg)
Success rate with/without image
stabilization
• Rule of 1/f
• Success rate
• 3-4 stops 8-16 times longer exposure
and size of convolution kernel ~ 4-8 pixels
![Page 38: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/38.jpg)
Hardware-based stabilization
summary
+ -Boosting ISO Cheap, almost no
additional hardware
Noisy image
Gyro-stabilized gimbals Universal, can stabilize
large motions
Heavy, expensive
OIS systems (Optical
image stabilization)
3-4 stops improvement High energy
consumption,
no „roll“ stabilization,
in all lenses – expensive
Moving sensor
stabilization
Roll stabilization, one
device for all lenses
![Page 39: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/39.jpg)
Talk outline
• How to describe the blur? (convolution,
velocity field, PSF )
• Hardware-based stabilization
• Software deblurring– Multiple underexposed/noisy images
– Non-blind restoration
– Single blurred image (deconvolution)
– Multiple blurred images (deconvolution)
– One blurred and one underexposed image
– Multiple images blurred by sideways vibrations
![Page 40: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/40.jpg)
underexposed = noisy
• Photon noise → SNR ~ SNR0* t
• increasing contrast amplifies noise
![Page 41: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/41.jpg)
Multiple noisy images
• Noise variance (and SNR) of the sum of N
images is the same as of the original image
• The difficult part is registration
N images
time t’=t/N
noise variance σ2/N
1 image
time t =1s
noise variance σ2
![Page 42: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/42.jpg)
Multiple noisy images
• Main problem slow read-out
• ¼ × 1/60s (15 times, ~4 stops)
15 images → 15*(1/3) = 5s
• Faster chips in near future allow avering of
4 - 8 images.
![Page 43: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/43.jpg)
Talk outline
• How to describe the blur? (convolution,
velocity field, PSF )
• Hardware-based stabilization
• Software deblurring– Multiple underexposed/noisy images
– Non-blind restoration
– Single blurred image (deconvolution)
– Multiple blurred images (deconvolution)
– One blurred and one underexposed image
– Multiple images blurred by sideways vibrations
![Page 44: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/44.jpg)
Restoration using known PSF
• Degradation model – for homogenous blur
u hu z
h
![Page 45: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/45.jpg)
Solution of deconvolution
problem
• Model
• 2 viewes
– Minimization of the model least squares error
(least squares fitting)
– Bayesian MAP estimation
![Page 46: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/46.jpg)
Minimization of LS error
• Image model
• Minimize
• Regularization constant - no one correct
value
![Page 47: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/47.jpg)
Role of regularization parameter
-5 -4.5 -4 -3.5 -3 -2.5 -2 -1.5 -1 -0.5 00
0.05
0.1
0.15
0.2
0.25
0.3
0.35
SNR = 15 dB, errmin = 0.073
SNR = 20 dB, errmin = 0.062
SNR = 30 dB, errmin = 0.045
- log
Mean least
squares error
/pixel
minu
![Page 48: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/48.jpg)
Matrix notation
u, z ... vectors
H ... matrix of 2D convolution
C ... regularization matrix
Tikhonov reg. c = [1 -1]
![Page 49: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/49.jpg)
Solution in Fourier domain
Parseval’s theorem
Convolution theorem
Wiener filter
Tikhonov reg. c = [1 -1]
![Page 50: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/50.jpg)
Bayesian view – MAP estimate
• MAP – Maximum a posteriori probability
• Maximize (using Bayes formula)
• Minimize
![Page 51: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/51.jpg)
Deconvolution as MAP estimate
• Minimize
![Page 52: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/52.jpg)
Image prior (first order statistics)
Intensity histogram Gradient log-histogram
![Page 53: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/53.jpg)
Equivalence of the two views
where and
Tikhonov regularization
![Page 54: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/54.jpg)
Image priors
Tikhonov regularization
TV regularization
![Page 55: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/55.jpg)
Space-variant deblurring
Minimization of
![Page 56: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/56.jpg)
Talk outline
• How to describe the blur? (convolution,
velocity field, PSF )
• Hardware-based stabilization
• Software deblurring– Multiple underexposed/noisy images
– Non-blind restoration
– Single blurred image (deconvolution)
– Multiple blurred images (deconvolution)
– One blurred and one underexposed image
– Multiple images blurred by sideways vibrations
![Page 57: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/57.jpg)
Single image deblurring - history
• Rob Fergus (2006) building on the work of
James Miskin
• Bayesian approach
• Approximation – conditional distributions
of PSF and image are considered
independent
• Priors on image gradients and blur kernels
as a mixture of Gaussians and exponential
functions
![Page 58: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/58.jpg)
Marginalization
• ln p(h|z) difficult to compute →
approximation
maxh
maxu,h
![Page 59: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/59.jpg)
Image prior
Gradient log-histogram
( approximation of ln p(∇ui) )
![Page 60: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/60.jpg)
Image priors
Tikhonov regularization
TV regularization
![Page 61: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/61.jpg)
Approximation by Gaussian mix
![Page 62: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/62.jpg)
PSF prior
![Page 63: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/63.jpg)
Rob Fergus (Example I)
![Page 64: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/64.jpg)
Rob Fergus (Example II)
![Page 65: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/65.jpg)
MAP approach at SIGGRAPH 08
![Page 66: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/66.jpg)
Single image deblurring -
summary
• Difficult, underdetermined problem
• Needs strong priors on both image and
convolution kernel
• First really successful algoritm (Fergus
2006) uses Bayesian variational approach,
priors are learned from example images
• MAP approaches less stable
• Hardly extensible to space-variant case
![Page 67: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/67.jpg)
Talk outline
• How to describe the blur? (convolution,
velocity field, PSF )
• Hardware-based stabilization
• Software deblurring– Multiple underexposed/noisy images
– Non-blind restoration
– Single blurred image (deconvolution)
– Multiple blurred images (deconvolution)
– One blurred and one underexposed image
– Multiple images blurred by sideways vibrations
![Page 68: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/68.jpg)
[u hk](x, y) + nk(x, y) = zk(x, y)
original image
Multiple blurred images
h1
hk
z1
zk
![Page 69: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/69.jpg)
Multi-image blind deconvolution
Energy minimization problem (well-posed)
System of integral equations (ill-posed,
underdetermined)
![Page 70: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/70.jpg)
Regularization terms
Q(u) =
![Page 71: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/71.jpg)
h2*u uh1 *= =z1 z2
z1 h2* *u h1= h2* z2 * h1h2u * =h1*
with one additional constraint
PSF regularization
![Page 72: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/72.jpg)
Incorporating a between-image
shift
)()())((][ yx,z=yx,n+yx,hu kkkk
)()()(][ yx,z=yx,n+yx,gu kkk
![Page 73: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/73.jpg)
Alternating minimization (AM)
AM of E(u,{gi}) over u and gi
Input: - blurred images
- estimation of the PSF size
Output: - reconstructed image
- the PSF’s
![Page 74: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/74.jpg)
Multiple blurred images
• Multichannel blind deconvolution
• Convolution model of blurring
• Solved by minimization of
![Page 75: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/75.jpg)
Multiple blurred images
![Page 76: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/76.jpg)
3-image deblurring (video)
![Page 77: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/77.jpg)
Multi-image deblurring -
summary
• Similar to methods used for single-image
deconvolution
• Much more data than in single-image case
→ we need less strong priors
• Can be applied to video
• In theory could be applied to space-variant
case, but slow
![Page 78: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/78.jpg)
Talk outline
• How to describe the blur? (convolution,
velocity field, PSF )
• Hardware-based stabilization
• Software deblurring– Multiple underexposed/noisy images
– Non-blind restoration
– Single blurred image (deconvolution)
– Multiple blurred images (deconvolution)
– One blurred and one underexposed image
– Multiple images blurred by sideways vibrations
![Page 79: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/79.jpg)
Blurred/underexposed - history
• 2006
– patented in US
– since 2006 - several papers assuming
convolution model
– simpler approach only match histograms, no
deconvolution
– Samsung introduced ASR (Advanced shake
reduction)
![Page 80: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/80.jpg)
Deblurring algorithm
Image registration
Blur kernel estimation
Space-variant restoration
Blurred
image
Noisy
image
![Page 81: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/81.jpg)
Image registration
• Small change of camera position – small
stereo base
• Static parts of the scene can be modelled
by projective tranform found by RANSAC
• Lens distortion can be neglected
• Less important parts of scene can move
![Page 82: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/82.jpg)
Blurred + underexposed results
![Page 83: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/83.jpg)
Blur kernel adjustment
• Regions lacking texture
• Regions of pixel saturation
![Page 84: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/84.jpg)
Restoration
• Minimization of functional
• PSF h interpolated from estimated
convolution kernels
![Page 85: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/85.jpg)
Shopping center (details)
![Page 86: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/86.jpg)
Bookcase example
![Page 87: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/87.jpg)
Bookcase (details)
![Page 88: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/88.jpg)
Shot-long exposure - summary
• fast and reliable
• works for space-variant blur
• potential for segmentation of moving
objects
• could be also extended to more images
![Page 89: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/89.jpg)
Talk outline
• How to describe the blur? (convolution,
velocity field, PSF )
• Hardware-based stabilization
• Software deblurring– Multiple underexposed/noisy images
– Non-blind restoration
– Single blurred image (deconvolution)
– Multiple blurred images (deconvolution)
– One blurred and one underexposed image
– Multiple images blurred by sideways vibrations
![Page 90: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/90.jpg)
In-plane translation
![Page 91: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/91.jpg)
How we compute camera trajectory
sensor plane
direction
of view
Existing methods Our method
Point traces (PSF) are scaled versions of camera
trajectory
Estimation of camera motion from the blurred
images is possible
sensor plane
direction
of view
![Page 92: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/92.jpg)
Algorithm removing motion blur
• 3 steps
• Explained on example images
• Algorithm for out-of-focus blur based on
similar principle but does not need step 1
![Page 93: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/93.jpg)
Estimation of camera motion (step
I)z1
z2
PSF consists of scaled
versions of camera trajectory
![Page 94: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/94.jpg)
Rough depth map estimation (step
II)
d0
z1
z2
![Page 95: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/95.jpg)
Functional minization (step III)
Input images z1, z2, …
Minimization initialized by depth map d0
Goal – sharp image and depth map
computed as argminu,d E(u,d)
![Page 96: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/96.jpg)
Functional minimization (step
III)z1
z2
![Page 97: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/97.jpg)
Motion blur + limited depth of
focusF/4
![Page 98: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/98.jpg)
Out-of-focus blur
(F/5.0)
(F/6.3)
F/16
z1
z2
![Page 99: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/99.jpg)
Software deblurring in present-
day cameras
• Usually no deblurring
• Samsung ASR system
– may use two images, one underexposed and
one blury - only simple algorithm, no
„deconvolution“
• Sony DSC-HX1 superimposes six photos
(update)
• Reason: speed and energy consumption
![Page 100: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/100.jpg)
Summary/Perspectives
• Denoising – readout speed problems – only way for now, limited EV improvement
• Single image approach – takes time, imprecise PSF, unable to distinguish intentional depth of focus, limited to convolution model
• Multiple blurred images – computationally expensive, fewer artifacts
• Blurred + underexposed image – relatively fast,but (so far) not enough to be used with real deblurring inside a camera
![Page 101: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/101.jpg)
Comparison with OIS
• Can remove roll motion (z-axis rotation)
blur
• Handle larger range of EV (exposure
values) but with growing number of
artifacts
• Ideal solution – both hardware and
software image stabilization
![Page 102: Recent advances in deblurring and image stabilization - avcr.czzoi.utia.cas.cz/files/ASI2008slides.pdf · Talk outline • How to describe the blur? (convolution, velocity field,](https://reader031.vdocument.in/reader031/viewer/2022022718/5c41c83293f3c338d7552295/html5/thumbnails/102.jpg)
Discussion, questions...
Michal Šorel
Academy of Sciences of the Czech Republic
www.zoi.utia.cas.cz