study dappled photography
DESCRIPTION
study of "Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing"TRANSCRIPT
Dappled Photography: Mask Enhanced Cameras forHeterodyned Light Fields and Coded Aperture Refocusing
Ashok Veeraraghavan, Ramesh Raskar, Amit AgrawalMitsubishi Electric Research Labs (MERL), Cambridge, MA
Ankit Mohan, Jack TumblinNorthwestern University, Evanston, IL
study
Abstract
• Heterodynes Light Field camera
• Coded Aperture camera
A theoretical framework for modulating 4D light fields using a mask between lens and sensors
Abstract
• Heterodynes Light Field camera
1. Add a high-frequency mask between Lens and Sensors
2. 4D light field - sense different rays from lens (u,v) in a sensor position (x,y)
3. Re-arrange light field
• Low resolution with different focus settings
• Full resolution in-focus
A theoretical framework for modulating 4D light fields using a mask between lens and sensors
Abstract
• Heterodynes Light Field camera
1. Add a high-frequency mask between Lens and Sensors
2. 4D light field - sense different rays from lens (u,v) in a sensor position (x,y)
3. Re-arrange light field
• Full resolution in-focus
• Low resolution with different focus settings
• Coded Aperture camera 1. Replace the Aperture
with a coded mask
2. A broadband mask enhance refocus at full resolution for Lambertian scene
A theoretical framework for modulating 4D light fields using a mask between lens and sensors
4D light field L(u,v,x,y)
Abstract
• Heterodynes Light Field camera
1. Add a high-frequency mask between Lens and Sensors
2. 4D light field - sense different rays from lens (u,v) in a sensor position (x,y)
3. Rearrange light field
• Low resolution with different focus settings
• Full resolution in-focus
A theoretical framework for modulating 4D light fields using a mask between lens and sensors
Abstract
• Heterodynes Light Field camera
1. Add a high-frequency mask between Lens and Sensors
2. 4D light field - sense different rays from lens (u,v) in a sensor position (x,y)
3. Re-arrange light field
• Low resolution with different focus settings
• Full resolution in-focus
• Coded Aperture camera 1. Replace the Aperture
with a coded mask
2. A broadband mask enhance refocus at full resolution for Lambertian scene
3. Refocusing partial
A theoretical framework for modulating 4D light fields using a mask between lens and sensors
Heterodynes Light Field camera
Heterodynes Light Field camera
• Based on modulation theorem in 4D frequency domain – mask carries rays
= rays x mask
F MaskRays
Modulation Theorem
• [Oppenheim et al. 99]http://en.wikipedia.org/wiki/Amplitude_modulation
Heterodynes Light Field camera
• Based on modulation theorem in 4D frequency domain – mask carries rays
= rays x mask
α depends on (d,v)
F MaskRays
good mask !
A poor mask blends the rays ! A good mask carriers the rays !
rays
Heterodynes Light Field camera
• Based on modulation theorem in 4D frequency domain – mask carries rays
= rays x mask
F MaskRays
Heterodynes Light Field camera
• Based on modulation theorem in 4D frequency domain – mask carries rays
• recover the light field by rearranging the tiles of 2D Fourier transfer into 4D plane to get the full resolution image information for the in-focus parts of the scene
F-1
= rays x mask
Rearrange
F MaskRays
Heterodynes Light Field camera
• Based on modulation theorem in 4D frequency domain – mask carries rays
• recover the light field by rearranging the tiles of 2D Fourier transfer into 4D plane to get the full resolution image information for the in-focus parts of the scene
• A raw sensor holds a modulated 4D light filed
= rays x mask
Raw sensor (modulate 4D light field
data)
In-focus at full resolution (demodulated)
Optical Heterodyning
Baseband Audio Signal
Receiver: DemodulationHigh Freq Carrier 100.1 MHz
ReferenceCarrier
Incoming Signal
Photographic Signal
(Light Field)
Carrier Incident Modulated
SignalReference
Carrier
Main LensObject Mask Sensor
RecoveredLight Field
Software Demodulation
99 MHz
Coded Aperture camera
• Base on Convolution• Aperture as a Modulator
▫ sinc function depends on θ
▫ Pinhole camera has a very very broadband modulator
▫ Design broadband mask
= rays x mask
Outline
•Introduction•Related Work•Theory & Framework•Heterodyne Light Field Camera•Encoded Blur Camera•Implements & Analysis•Contributions & Future Work
Introduction
Light Field
http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt
SensorLens
θ x
Imaginary film
x
θ
Sensed image (in-focus)
object
object
red : the in-focus lineyellow : sample
http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt
Lens Sensor
θ x
Imaginary film
x
θ
Sensed image (in-focus)
http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt
Lens Sensor
u x
Imaginary film
x
θ
Sensed image (in-focus)red : the in-focus lineyellow : sample
http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt
Lens Sensor
θ x
x
θ
Imaginary film
Sensed image (out of focus, far)red : the in-focus lineyellow : sample
http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt
Lens Sensor
θ x
x
θ
Imaginary film
Sensed image (out of focus, far)red : the in-focus lineyellow : sample
http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt
Lens Sensor
θ x
x
θ
Imaginary film
Sensed image (out of focus, near)red : the in-focus lineyellow : sample
http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt
Lens Sensor
θ x
x
θ
Imaginary film
Sensed image (out of focus, near)red : the in-focus lineyellow : sample
Light Field Acquisition
Integral Photography• Interal Photograpy
[Lippmann 1908]
• Integral camera [Okano et al. 99; Martnez-Corral et al. 04; Javidi and Okano 02]
• Light field Camera▫ Virtual viewpoint
[Levoy and Hanrahan 96]
[Gertler et al 96]
▫ Virtual aperture [Levoy and Hanrahan
96] [Isaksen et al. 00]
▫ Synthetic appearture photography (similar virtual aperture) [Levoy et al. 04] [Vaish et al. 04]
Light field Camera
Plenoptic cameraLight field rendering Dapped Photography
• [Levoy and Hanrahan Siggraph 96]
• [Gortler et al 96, 06]
• [Adelson et al, IEEE95]
• [Levoy and Hanrahan Siggraph 96]
• [Gortler et al 96]
• Hand-held light field camera [R Ng et al 05]
• Fourier slice photography [R Ng, SIGGRAPH05]
• The mask weights the rays
Related work
• Coded Imaging▫ Coded aperture imaging
Overcome the limit of pinhole camera [Skinner 98]
▫ Coded Exposure Camera [Raskar et al. 06]
• Wavefront Coding [Dowski and Cathey 95; Dowski and Johnson 99; van der
Gracht et al. 96] Traditional lens based [Farid and Simoncelli 98] Both wavefront and coded aperture [Jahnson et al. 00]
• Deblurring & deconvolution Include extended DOF images by refocusing a light field at
multiple depth and applying the digital photomontage tech. [Agarwala et al. 04]
Fusion of multiple blurred images [ Jaeberli 94]
Theory & Framework
For different focus settings, the obtained images correspond to slices at different angles, “Fourier Slice Photography ” [Ng, R. 05]
Assumption : simulate the aperture as mask placed at lens
Open Aperture
Assumption : a planar Lambertian object at the focus plane
Because no angular variations in the irradiance of rays from a Lambertian object, the content of light field is restricted to be along the fx axis
The sensed image is a slice of the modulated light field
Open Aperture
• In-focus sensor
▫ The in-focus image corresponds to a slice of LA(fx, f θ) along fx (f θ =0)
▫ No information lost
• Out of focus sensor▫ The sensor image is a slanted slice
▫ The slant angle depends on the degree of mis-focus
Open Aperture
Heterodyne Light Field Camera
Mask as Modulator
Mask as Modulator
• d = v (at aperture stop, θ plane)▫ Mask affects the all rays at an angle θ in a similar way !▫ m(x, θ) = c (y = θ)▫ α = 900
• d = 0 (at sensor, conjugate plane)▫ Mask attenuates all rays for the same x equally !▫ m(x, θ) = c (y = x)▫ α = 00
Mask as Modulator
• Optimal Mask Position
▫ In practice, since the spatial resolution is much larger than the angular resolution, is very small, and therefore the mask needs to be placed close to the sensor
• Optimal Mask Pattern
•Harmonic sine wave
•Boost
Notes
• 4D light field
• Aliasing▫ When band-limit assumption is not valid in the spatial
dimension, the energy in the higher spatial frequencies of the light field masquerade as energy in the lower angular dimension.
▫ Post-filter the recovered light field using a Kaiser-Bessel filter with a filer width of 1.5 [Ng 05]
Encoded Blur Camera
Mask as modulator
Assumption : layered Lambertian scene
Because no angular variations in the irradiance of rays from a Lambertian scene, the content of light field is restricted to be along the fx axis
∵
Optimal Mask for Encoding Defocus Blur
•Blurred image is linear convolution (circularly convolution with zero padded)▫Defocus by PSF (point spread function)
•Coded aperture remove SNR only special cases
+ Star , -Natural photography
- Optimal mask – continuous valued code by gradient decent optimization (Matlab, fmincon)- 7x7 Binary mask as initial guess
- 10 hours of search
Implementation & Analysis
Light Filed Camera
Mask & Sensor
Heterodyne Light Field Camera210 mm f/5.6Nikkor-W Lens
CanoScan LiDE 70scanner sensor
Mask80 dots/mm
Raw sensor image
Scene parts which are in-focus can be recovered at full resolution
Far Focused
Near Focused
In-focus – full resolution Out of focus Low resolution refocused image
Analysis
- Scanner sensor leading to pattern noise (horizontal /vertical lines)
+ Easy to cover over in a conventional digital camera with a finer mask placed inside in the future
• Computation
+ Computation burden is low because of computing light field and refocusing is done in Fourier domain
- Calibration of in-plane rotation and shift of the mask with respect to sensor
Failure Cases
• If Assumption of a band-limited light field is invalid, the aliasing artifacts in recovered light field
• 2D cosine mask needs to be moved away from the sensor because it results in diffraction
Encoded Blur Camera
Low resolution Mask
Encoded Blur Camera100 mm f/2.8 USM Macro Lens Mask Sensor
Canon Rebel XT SLR camera
ISO-12233 Chart
Modulation Transfer Function (MTF) of ISO-12233
MTF: low MTF: high
Full resolution digital refocusing using encoded blur camera
Captured photo
Refocused photo
In-focus fence + blurred person
Deblurring without taking the occluders
into account
Weighted deconvolution Eq.
Binary mask for the occluders
• In this case, we can recover the sharp image if the blur size is larger than the occluder size
• b is the vectorized blurred image
• A is the block-Toeplitz matrix representing 2D blur
• W is a weighting matrix which sets the weights corresponding to the occluded pixels in the blurred image to zero
Failure Cases
• Scenes with large variation in depths and those with view dependencies can not be handle
▫ Practice value
7x7 mask : blur size of about 20 pixels
• Finer resolution mask can handle large defocus blur but lead to diffraction blur
Contributions
• A theoretical framework of modulating 4D light fields camera working on frequency domain
• A new class of 4D light filed camera holds full resolution modulated 4D light field
• Don’t require additional optical elements such as lens arrays
• Analyze defocus blur as a special case of the frequency domain re-mapping and demonstrate that a broadband mask at aperture can preserve high spatial frequencies in defocused image
= rays x mask
Future Work
• Light Fields for Dynamic Scenes▫ Changing masks with time▫ Coding in time and space
• General Ray Modulators▫ Tilted/curved/multiple masks▫ Wavelength dependent masks▫ Angular/Spatial Resolution Tradeoff
• Applications▫ Estimating lens aberration▫ Microscopy▫ Light Field Applications