![Page 1: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/1.jpg)
Near-Invariant Blur for Depth and 2D Motion
via Time-Varying Light Field Analysis
Yosuke Bando1,2 Henry Holtzman2 Ramesh Raskar2
1Toshiba Corporation 2MIT Media Lab
![Page 2: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/2.jpg)
Defocus & Motion Blur
PSF
![Page 3: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/3.jpg)
Depth and Motion-Invariant Capture
PSF
![Page 4: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/4.jpg)
Deblurring Result
![Page 5: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/5.jpg)
Outline
• Motivation
• Related Work
• Intuitions
• Analysis
• Results
• Conclusions
![Page 6: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/6.jpg)
Outline
• Motivation
• Related Work
• Intuitions
• Analysis
• Results
• Conclusions
![Page 7: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/7.jpg)
Joint Defocus & Motion Deblurring
Image
capture
Local blur
estimation
Non-uniform
deblurring
Standard approach
• Estimate depth and motion from a single image
• Recover lost high-frequency content
Extremely difficult
![Page 8: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/8.jpg)
Joint Defocus & Motion Deblurring
Image
capture
Local blur
estimation
Non-uniform
deblurring
Standard approach
Uniform
deconvolution
Depth and 2D motion-
invariant image capture
Proposed approach
No blur estimation
Well-studied problem
![Page 9: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/9.jpg)
Outline
• Motivation
• Related Work
• Intuitions
• Analysis
• Results
• Conclusions
![Page 10: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/10.jpg)
Depth-Invariant Capture
Spectral focus sweep
[Cossairt and Nayar 2010]
Wavefront coding
[Dowski and Cathey 1995]
Focus sweep
[Hausler 1972;
Nagahara et al. 2008]
Diffusion coding
[Cossairt et al. 2010]
Depth-invariant image
Deblurred
![Page 11: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/11.jpg)
1D Motion-Invariant Capture
• Invariant to object speed
• Motion direction must be fixed
– Horizontal, for example
Image
sensor
accelerate
Motion-invariant image Deblurred Normal camera
[Levin et al. 2008]
![Page 12: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/12.jpg)
Computational Cameras for Deblurring
Defocus deblurring Motion deblurring
High frequency
preservation
(non-invariant)
Coded aperture
[Levin et al. 2007;
Veeraraghavan et al. 2007]
Lattice-focal lens
[Levin et al. 2009]
Coded exposure
[Raskar et al. 2006]
Orthogonal parabolic exposures
[Cho et al. 2010]
Circular sensor motion
[Bando et al. 2011]
Invariant capture Wavefront coding
[Dowski and Cathey 1995]
Focus sweep
[Hausler 1972;
Nagahara et al. 2008]
Diffusion coding
[Cossairt et al. 2010]
Spectral focus sweep
[Cossairt and Nayar 2010]
Motion-invariant photography
(for 1D motion)
[Levin et al. 2008]
Also nearly
2D motion-invariant
No joint defocus and motion deblurring
No 2D motion-invariant capture
![Page 13: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/13.jpg)
Outline
• Motivation
• Related Work
• Intuitions
• Analysis
• Results
• Conclusions
![Page 14: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/14.jpg)
Depth-Invariance for Static Point
Sensor Aperture Plane of focus
Scene
point
Time
![Page 15: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/15.jpg)
Depth-Invariance for Static Point
Sensor Aperture Plane of focus
Scene
point
Time
![Page 16: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/16.jpg)
Depth-Invariance for Static Point
Sensor Aperture Plane of focus
Scene
point
Time
![Page 17: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/17.jpg)
Motion-Invariance for Moving Point
Sensor Aperture
Scene
point
Motion
Time
![Page 18: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/18.jpg)
Follow Shot
http://commons.wikimedia.org/wiki/File:Bruno_Senna_2006_Australian_Grand_Prix-3.jpg
![Page 19: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/19.jpg)
Follow Shots for Various Motions
x
t
y
![Page 20: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/20.jpg)
Follow Shots for Various Motions
x
t
y
![Page 21: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/21.jpg)
Follow Shots for Various Motions
x
t
y
![Page 22: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/22.jpg)
Outline
• Motivation
• Related Work
• Intuitions
• Analysis
• Results
• Conclusions
![Page 23: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/23.jpg)
Analysis
• Photo is a projection of a light field [Ng 2005]
𝐷 𝐱0 = 𝑘 𝐱0 − 𝐱,−𝐮 ⋅ 𝑙 𝐱, 𝐮 𝑑𝐱𝑑𝐮
Light field
kernel
Light field
x = (x, y)
u = (u, v) Defocus-blurred
image
x
y
u
v Sensor
Aperture
Scene point
b
A
![Page 24: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/24.jpg)
Analysis
• Photo is a projection of a time-varying light field
𝐷 𝐱0 = 𝑘 𝐱0 − 𝐱,−𝐮 ⋅ 𝑙 𝐱, 𝐮 𝑑𝐱𝑑𝐮
Light field
kernel
Light field
x = (x, y)
u = (u, v) Defocus-blurred
image
x
y
u
v Sensor
Aperture
Scene point
b
A Velocity
(mx, my)
![Page 25: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/25.jpg)
Analysis
• Photo is a projection of a time-varying light field
x
y
u
v Sensor
Aperture
Scene point
b
A Velocity
(mx, my)
𝐷 𝐱0 = 𝑘 𝐱0 − 𝐱,−𝐮,−𝑡 ⋅ 𝑙 𝐱, 𝐮, 𝑡 𝑑𝐱𝑑𝐮𝑑𝑡 x = (x, y)
u = (u, v) Time-varying
light field kernel
Time-varying
light field
Defocus/motion-
blurred image
![Page 26: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/26.jpg)
Time-Varying Light Field Analysis
• Photo is a projection of a time-varying light field
𝜙𝑠,𝐦 𝐱 = 𝑘 𝐱 + 𝑠𝐮 + 𝐦𝑡, 𝐮, 𝑡 𝑑𝐮𝑑𝑡
Lambertian scene at depth s with velocity m = (mx, my)
Joint defocus &
motion blur PSF Magnitude of 2D Fourier transform
Modulation transfer function (MTF)
|𝜙 𝑠,𝐦 𝐟𝐱 |2 = |𝑘 𝐟𝐱, −𝑠𝐟𝐱, −𝐦 ⋅ 𝐟𝐱 |2
𝐷 𝐱0 = 𝑘 𝐱0 − 𝐱,−𝐮,−𝑡 ⋅ 𝑙 𝐱, 𝐮, 𝑡 𝑑𝐱𝑑𝐮𝑑𝑡 x = (x, y)
u = (u, v) Time-varying
light field kernel
Time-varying
light field
Defocus/motion-
blurred image
![Page 27: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/27.jpg)
Analysis Procedure and Findings
• For each existing computational camera for deblurring
– 1. derive a kernel equation describing the optical system
– 2. calculate its Fourier transform to obtain the MTF
– 3. compare it with the theoretical upper bounds
58% 66% Better than any other existing computational cameras for deblurring
![Page 28: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/28.jpg)
Outline
• Motivation
• Related Work
• Intuitions
• Analysis
• Results
• Conclusions
![Page 29: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/29.jpg)
Prototype Focus Sweep Camera
![Page 30: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/30.jpg)
Prototype Camera & Setup
Hot shoe
Arduino
+ batteries
SPI command to move focus
Shutter release signal
Reference camera
Focus sweep camera
Beam splitter
Scene
![Page 31: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/31.jpg)
Normal Camera Image
Focused Defocused
Motion blur Motion blur
![Page 32: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/32.jpg)
Focus Sweep Image
![Page 33: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/33.jpg)
Deconvolution Result
![Page 34: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/34.jpg)
Short Exposure Narrow Aperture Image
![Page 35: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/35.jpg)
More Examples
Focus
Motion
Standard camera Focus sweep Deconvolution results
N/A
![Page 36: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/36.jpg)
Limitations
• Object depth and speed ranges must be bounded
• Depth and speed ranges cannot be adjusted separately
• Object motion must be in-plane linear
• Camera shake cannot be handled
Standard camera Focus sweep Deconvolved
![Page 37: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/37.jpg)
Standard camera Focus sweep Deconvolution results
Rotation & Z Motion
Focus
Motion
Focus
Motion
![Page 38: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/38.jpg)
Summary
• Simple approach to joint defocus & motion deblurring
– No need for estimating scene depth or motion
– Also preserves high-frequency image content
– Theoretically near-optimal
– Has practical implementation (just firmware update)
Standard camera Focus sweep Deconvolution results
![Page 39: Near-Invariant Blur for Depth and 2D Motion via …web.media.mit.edu/~bandy/invariant/SIG13invariant_talk.pdfVeeraraghavan et al. 2007] Lattice-focal lens [Levin et al. 2009] Coded](https://reader036.vdocument.in/reader036/viewer/2022071016/5fcf210322a5d5478f18b194/html5/thumbnails/39.jpg)
Summary
• Simple joint defocus & motion deblurring
– No depth or motion estimation
– Preserves high-frequency
– Theoretically near-optimal
– Practical implementation
• http://www.media.mit.edu/~bandy/invariant/
– How to control the lens
– How to achieve perfect invariance
• Computational Cameras & Displays 2013
• Yusuke Iguchi
• Noriko Kurachi
• Matthew Hirsch,
• Matthew O’Toole
• Douglas Lanman
• Cheryl Sham
• Sonia Chang
• Shih-Yu Sun
• Jeffrey W. Kaeli
• Bridger Maxwell
• Austin S. Lee
• Saori Bando
Acknowledgments