coded aperture and optical heterodyning : a mask …raskar/mask/raskarmaskslides.pdfcomputing 4d...
TRANSCRIPT
![Page 1: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/1.jpg)
Coded aperture and Optical Heterodyning : A mask-based approach for Digital Refocusing and Light
Field Acquisition by Conventional Cameras
![Page 2: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/2.jpg)
Heterodyne Light Field Camera
![Page 3: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/3.jpg)
Consider a 2D camera that can capture 4D light fields ..
2D Sensor image Video showing refocusing from 4D light field
![Page 4: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/4.jpg)
.. as well as high resolution image of focused parts of scene ..
High resolution image of in focus parts of the scene
![Page 5: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/5.jpg)
.. using only a mask placed inside the camera and
no additional optics
Main LensObject Mask Sensor
![Page 6: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/6.jpg)
Results
2D Sensor image
Zoom in showing the effect of mask
![Page 7: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/7.jpg)
Computing 4D Light Field
2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052
FFT
Reshape 2D tiles into 4D planes181*228*9*94D IFFT
4D Light Field
9*9=81 such tiles
181*228*9*9
![Page 8: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/8.jpg)
Refocusing: Take slices of 4D Fourier transform2D Fourier Transform
2D FFT
Reshape 2D tiles into 4D planes
Take a 2D Slice
2D IFFT181*228
refocused image
181*228*9*9
2D Sensor image, 1629*2052
![Page 9: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/9.jpg)
Demonstrating parallax between two of the views
![Page 10: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/10.jpg)
Refocusing of transparent layers
2D sensor image. The scene consist of a transparent glass sheet in front with text written on it.
![Page 11: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/11.jpg)
Refocusing of transparent layers
High resolution image of the focused parts of the scene
![Page 12: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/12.jpg)
Refocusing of transparent layers
Low resolution refocused image on the glass sheet
in front
![Page 13: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/13.jpg)
Encoded Blur Camera for Extended Depth of Field
(for Layered Lambertian Scenes)
![Page 14: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/14.jpg)
We analyze out of focus deblurring in 4D Fourier space
Main Lens S0Object
x-plane θ -plane
l(xi,θ0)l(xi,θj)
l(xi,-θ0)
Conventional Camera Plane of focus
![Page 15: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/15.jpg)
In Fourier domain, the image is a slice of light field
Plane of focus
fx
fθL(fx,fθ)
LM(fx,fθ)Sensor Slice
Light field of a Lambertian scene
Open Aperture
M(fx,fθ)
fx
fθ
fx
fθ
Modulated Light Field
Main Lens S0Object
x-plane θ -plane
l(xi,θ0)l(xi,θj)
l(xi,-θ0)
Conventional Camera
Snear Sfar
![Page 16: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/16.jpg)
By putting a mask at the aperture, the aperture modulation function is replaced by the mask modulation function
fx
fθL(fx,fθ)
LM(fx,fθ)Out of focus Sensor Slice
Light field of a Lambertian scene
Mask
fx
fθ
fx
fθ
Modulated Light Field
High Frequencies are preserved if broadband mask is used
![Page 17: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/17.jpg)
Digital Refocusing
Captured Blurred Image
![Page 18: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/18.jpg)
Digital Refocusing
Refocused Image on Person
![Page 19: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/19.jpg)
Digital Refocusing
![Page 20: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/20.jpg)
Digital Refocusing
Captured Blurred Image
![Page 21: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/21.jpg)
Digital Refocusing
Refocused Image on Person
![Page 22: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/22.jpg)
Deblurring in presence of partial occluders
Captured Photo Mask for occluders
Least Squares Deblurring
Weighted Least Squares Deblurring
![Page 23: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/23.jpg)
Comparison with Small Aperture Image
Captured Blurred Image
Small Aperture Image
Deblurred Image
![Page 24: Coded aperture and Optical Heterodyning : A mask …raskar/Mask/raskarMaskSlides.pdfComputing 4D Light Field 2D Sensor image, 1629*2052 2D Fourier Transform, 1629*2052FFT Reshape 2D](https://reader035.vdocument.in/reader035/viewer/2022062922/5f097e987e708231d427183d/html5/thumbnails/24.jpg)
Comparison with Traditional Camera
Captured Blurred Image
Deblurred Image
Encoded Blur Camera Traditional Camera