Motion Deblurring Using Hybrid Imaging

Moshe Ben-Ezra and Shree K. Nayar Columbia University

IEEE CVPR ConferenceJune 2003, Madison, USA

Image Recording Requires Time

Niépce 18278 hours exposure

Daguerre 18291/2 hour exposure

Motion Blur is Everywhere

Object Motion

Camera Motion

Stabilized Lenses

1/250 second (< -1 stop) Stabilization drifts with time Rotation only

1/15 second (< -5 stops)

Canon Stabilized lens 400mm

Blind Image Deconvolution

Accurate Point Spread Function (PSF) Needed.

Motion Point Spread Function (PSF)

Motion PSF is a Function of:

1. Motion path2. Motion speed

X

Y

Ener

gy ~

1/ s

peed

Spatial spread

H

PSF Detector?

Camera

PSF Detector

Can the PSF detector be a small and simple

imaging device ?

Electron wells

Fundamental Limits of Imaging

Detector’s noise level

Photon flux

Detector

Pixel’s Signal Noise

Fundamental Resolution Tradeoff

Spatial resolution (pixels)

Tem

pora

l res

olut

ion

(fps

)

30

330K720x480

Conventional video camera

130

3

3M2048x1536

Hi-resolution camera

75K320x240

Low-resolution camera

Hybrid imaging system

A Hybrid camera enjoys both worlds

Overview of Approach

PSF Estimation

Low-Res. camera

Hi-Res. camera

Same time period

Deconvolution

Motion Analysis

x

y

Global Motion From Low Resolution Detector

uv1

cos sin x sin cos y

0 0 1

xy1

TranslationRotation

2

),(minarg

tI

yIv

xIu

vu

Objective function (Optical flow constraint)

Lucas Kanade

Simulations: Motion Accuracy from Low- Res. Images

NoiseResolution

= 3 = 9 = 27 = 81

640x640 (1:1) 0.01 0.01 0.02 0.04320x320 (1:4) 0.03 0.04 0.05 0.1

160x160 (1:16) 0.03 0.04 0.07 0.480x80 (1:64) 0.13 0.21 0.39 2.6

Average Motion Error in Pixels

Constraints on Continuous PSF

Energy conservation constraint:

yx

dydxyxh 1),(

h(x(t),y(t))dt ttend tstartt

t t

Path is continuous and twice differentiable

Constant flux assumption:

Smoothness constraint:

PSF Estimation from Computed Motion

x

f1f2 f3 f4

f5

f6y

Frame 2 … Frame 5

y f1f2 f3 f4

f5

f6

x

yh

h2

h3

h4h5

Frame 2 … Frame 5

yh

h2

h3

h4h5

Frame 2 … Frame 5

x

Deconvolution of High Resolution Image Standard iterative ratio-based algorithm*

,ˆ)()()(ˆ)(ˆ

)()(ˆ

)()()1(

)0(

k

kk

OSxIxSxOxO

xIxO

Guaranties non-negative pixel result

* Richardson [72] Lucy [74]

ErrorPSFImage estimate

Designs for Hybrid Imaging

A rig of two cameras

Using a special chip

Using a beam splitter

Our Prototype: Rig of Two Cameras

Primary detector(2048x1536)

Secondary detector(360x240)

Resolution ratio of 1 : 36

Example 1 - Blurred Hi-Res Image

f = 633mm, Exp. Time 1 Sec (> -9 stops)

PSF Estimation from Motion

Low resolution sequence.

X (Pixels)10 130

10

90

Y (P

ixel

s)0.001

0.06

Estimated PSF

f = 633mm, Exp. Time 1 Sec

Deblurred Image

f = 633mm, Exp. Time 1 Sec

Example 1 - Comparison

Deblurred imageBlurred image f = 633mm, Exp. Time 1 Sec

Tripod image (Ground Truth)

Example 2 - Blurred Night Image

f = 884mm, Exp. Time 4 Sec (> -11 stops)

PSF Estimation from Motion

X (Pixels)10 60

10

30

Y (P

ixel

s)0.001

0.003

f = 884mm, Exp. Time 4 Sec

Low resolution sequence.

Deblurred Night Image

f = 884mm, Exp. Time 4 Sec

Example 3 - Comparison

Deblurred imageBlurred image

Tripod image (Ground Truth)

f = 884mm, Exp. Time 4 Sec

Object Deblurring Problem

Moving objects blend into the background

Hybrid Imaging Solution (simulated)

Requires clear high-resolution background image

Quantifying The Affect of Motion Blur

Empirical tests: RMS error. Volume of Solutions (Linear Model):

y Ax z x A 1y A 1z

High-ResolutionImage

Uncertainty (Quantization)

InputImages

Volume of Solutions 1/det(A)

BlurDecimation