fourier slice photography ren ng stanford university
TRANSCRIPT
Fourier Slice Photography
Ren Ng
Stanford University
Conventional Photograph
Light Field Photography
Capture the light field inside the camera body
Hand-Held Light Field Camera
Medium format digital camera Camera in-use
16 megapixel sensor Microlens array
Light Field in a Single Exposure
Light Field in a Single Exposure
Light Field Inside the Camera Body
Digital Refocusing
Digital Refocusing
Questions About Digital Refocusing
What is the computational complexity?Are there efficient algorithms?
What are the limits on refocusing?How far can we move the focal plane?
Overview
Fourier Slice Photography Theorem
Fourier Refocusing Algorithm
Theoretical Limits of Refocusing
Previous Work
Integral photography
Lippmann 1908, Ives 1930
Lots of variants, especially in 3D TVOkoshi 1976, Javidi & Okano 2002
Closest variant is plenoptic cameraAdelson & Wang 1992
Fourier analysis of light fields
Chai et al. 2000
Refocusing from light fields
Isaksen et al. 2000, Stewart et al. 2003
Fourier Slice Photography Theorem
In the Fourier domain, a photograph is a 2D slice in the 4D light field.
Photographs focused at different depths correspond to 2D slices at different trajectories.
Digital Refocusing by Ray-Tracing
Lens Sensor
u x
Digital Refocusing by Ray-Tracing
Lens Sensor
u
Imaginary film
x
Digital Refocusing by Ray-Tracing
Lens Sensor
u x
Imaginary film
Digital Refocusing by Ray-Tracing
Lens Sensor
u
Imaginary film
x
Digital Refocusing by Ray-Tracing
Lens Sensor
u x
Imaginary film
Refocusing as Integral Projection
Lens Sensor
u x
Imaginary film
x
u
Refocusing as Integral Projection
Lens Sensor
u x
Imaginary film
x
u
Refocusing as Integral Projection
Lens Sensor
u x
x
u
Imaginary film
Refocusing as Integral Projection
Lens Sensor
u x
x
u
Imaginary film
Classical Fourier Slice Theorem
2D FourierTransform
1D FourierTransform
IntegralProjection
Slicing
2D FourierTransform
Classical Fourier Slice Theorem
1D FourierTransform
IntegralProjection
Slicing
Classical Fourier Slice Theorem
2D FourierTransform
1D FourierTransform
IntegralProjection
Slicing
Fourier Domain
Classical Fourier Slice Theorem
IntegralProjection
Slicing
Spatial Domain
Spatial Domain
Classical Fourier Slice Theorem
IntegralProjection
Slicing
Fourier Domain
Fourier Slice Photography Theorem
IntegralProjection
Slicing
Fourier Domain
Spatial Domain
Fourier Slice Photography Theorem
4D FourierTransform
IntegralProjection
Slicing
Fourier Slice Photography Theorem
4D FourierTransform
2D FourierTransform
IntegralProjection
Slicing
Fourier Slice Photography Theorem
4D FourierTransform
2D FourierTransform
IntegralProjection
Slicing
Fourier Slice Photography Theorem
4D FourierTransform
2D FourierTransform
IntegralProjection
Slicing
Photographic Imaging Equations
Spatial-Domain Integral Projection
Fourier-Domain Slicing
Photographic Imaging Equations
Spatial-Domain Integral Projection
Fourier-Domain Slicing
Photographic Imaging Equations
Spatial-Domain Integral Projection
Fourier-Domain Slicing
Theorem Limitations
Film parallel to lens
Everyday camera, not view camera
Aperture fully open
Closing aperture requires spatial mask
Overview
Fourier Slice Photography Theorem
Fourier Refocusing Algorithm
Theoretical Limits of Refocusing
Existing Refocusing Algorithms
Existing refocusing algorithms are expensive
O(N4) where light field has N samples in each dimension
All are variants on integral projection Isaksen et al. 2000 Vaish et al. 2004 Levoy et al. 2004 Ng et al.
2005
Refocusing in Spatial Domain
4D FourierTransform
2D FourierTransform
IntegralProjection
Slicing
Refocusing in Fourier Domain
4D FourierTransform
Inverse2D FourierTransform
IntegralProjection
Slicing
Refocusing in Fourier Domain
4D FourierTransform
Inverse2D FourierTransform
IntegralProjection
Slicing
Asymptotic Performance
Fourier-domain slicing algorithm
Pre-process: O(N4 log N)
Refocusing: O(N2 log N)
Spatial-domain integration algorithm
Refocusing: O(N4)
Resampling Filter Choice
Triangle filter (quadrilinear)
Kaiser-Bessel filter(width 2.5)
Gold standard (spatial integration)
Overview
Fourier Slice Photography Theorem
Fourier Refocusing Algorithm
Theoretical Limits of Refocusing
Problem Statement
Assume a light field camera with
An f /A lens
N x N pixels under each microlens
If we compute refocused photographs from these light fields, over what range can we move the focal plane?
Analytical assumption
Assume band-limited light fields
Band-Limited Analysis
Band-Limited Analysis
Light field shotwith camera
Band-width of measured light field
Band-Limited Analysis
Band-Limited Analysis
Band-Limited Analysis
Photographic Imaging Equations
Spatial-Domain Integral Projection
Fourier-Domain Slicing
Results of Band-Limited Analysis
Assume a light field camera with An f /A lens N x N pixels under each microlens
From its light fields we can Refocus exactly within
depth of field of an f /(A � N) lens
In our prototype camera Lens is f /4 12 x 12 pixels under each microlens
Theoretically refocus within depth of field of an f/48 lens
Light Field Photo Gallery
Stanford Quad
Rodin’s Burghers of Calais
Palace of Fine Arts, San Francisco
Palace of Fine Arts, San Francisco
Waiting to Race
Start of the Race
Summary of Main Contributions
Formal theorem about relationship between light fields and photographs
Computational application gives asymptotically fast refocusing algorithm
Theoretical application gives analytic solution for limits of refocusing
Future Work
Apply general signal-processing techniques
Cross-fertilization with medical imaging
Thanks and Acknowledgments
Collaborators on camera tech report Marc Levoy, Mathieu Brédif, Gene
Duval, Mark Horowitz and Pat Hanrahan
Readers and listeners Ravi Ramamoorthi, Brian Curless,
Kayvon Fatahalian, Dwight Nishimura, Brad Osgood, Mike Cammarano, Vaibhav Vaish, Billy Chen, Gaurav Garg, Jeff Klingner
Anonymous SIGGRAPH reviewers
Funding sources NSF, Microsoft Research Fellowship,
Stanford Birdseed Grant
Questions?
“Start of the race”, Stanford University Avery Pool, July 2005