focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10....
TRANSCRIPT
![Page 1: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/1.jpg)
Focal stacks and lightfields
15-463, 15-663, 15-862Computational Photography
Fall 2017, Lecture 16http://graphics.cs.cmu.edu/courses/15-463
![Page 2: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/2.jpg)
Course announcements
• Homework 4 is out.- Due October 26th.- Bilateral filter will take a very long time to run.- Drop by Yannis’ office to pick up cameras any time.
• Project ideas were due on Piazza on Friday 20th.- Responded to most of you.- Some still need to post their ideas.
• Project proposals are due on Monday 30th.
![Page 3: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/3.jpg)
Overview of today’s lecture
• Pinhole vs lens cameras.
• Focal stack.
• Lightfield.
• Measuring lightfields.
• Plenoptic camera.
• Images from lightfields.
![Page 4: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/4.jpg)
Slide credits
Most of these slides were adapted from:
• Fredo Durand (MIT).• Gordon Wetzstein (Stanford).
![Page 5: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/5.jpg)
Pinhole vs lens cameras
![Page 6: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/6.jpg)
Pinhole camera
• Everything is in focus.• Very light inefficient.
![Page 7: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/7.jpg)
Lens camera
• Only one plane is in focus.• Very light efficient.
How can we get an all in-focus image?
focus plane
![Page 8: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/8.jpg)
Focal stack
![Page 9: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/9.jpg)
Focal stack
• Capture images focused at multiple planes.• Merge them into a single all in-focus image.
Analogous to what we did in HDR• Focal stack instead of exposure stack.
![Page 10: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/10.jpg)
Homework 5: focal stack imaging
![Page 11: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/11.jpg)
Focal stack imaging1. Capture a focal stack
2. Merge into an all in-focus image
![Page 12: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/12.jpg)
Focal stack imaging1. Capture a focal stack
2. Merge into an all in-focus image
![Page 13: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/13.jpg)
How do you capture a focal stack?
Which of these parameters would you change (and how)?
lens-sensor distance D’lens-object distance D
lens aperture f/#
lens focal length f
![Page 14: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/14.jpg)
How do you capture a focal stack?
Which of these parameters would you change (and how would you achieve that)?
lens-sensor distance D’lens-object distance D
lens aperture f/#
lens focal length f rotate lens focus ring
(not zoom!)
![Page 15: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/15.jpg)
How do you capture a focal stack?
Which of these parameters would you change (and how would you achieve that)?
lens-sensor distance D’lens-object distance D
lens aperture f/#
lens focal length f
![Page 16: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/16.jpg)
Capturing a focal stack
In-focus plane in each stack image
![Page 17: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/17.jpg)
Focal stack imaging1. Capture a focal stack
2. Merge into an all in-focus image
![Page 18: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/18.jpg)
How do you merge a focal stack?
![Page 19: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/19.jpg)
How do you merge a focal stack?1. Align images2. Assign per-pixel weights representing “in-focus”-ness3. Compute image average
![Page 20: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/20.jpg)
How do you merge a focal stack?1. Align images2. Assign per-pixel weights representing “in-focus”-ness3. Compute image average
![Page 21: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/21.jpg)
Image alignmentWhy do we need to align the images?
![Page 22: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/22.jpg)
Image alignmentWhy do we need to align the images?• When we change focus distance, we also change field of view (magnification).• Also, scene may not be static (but we will be ignoring this for now).
change in magnification
change in focus
![Page 23: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/23.jpg)
Image alignment
![Page 24: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/24.jpg)
Image alignment
![Page 25: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/25.jpg)
Image alignmentWhy do we need to align the images?• When we change focus distance, we also change field of view (magnification).• Also, scene may not be static (but we will be ignoring this for now).
change in magnification
change in focus
• Assume we know f and all D’ values. • How do can we align the images?
![Page 26: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/26.jpg)
Image alignmentWhy do we need to align the images?• When we change focus distance, we also change field of view (magnification).• Also, scene may not be static (but we will be ignoring this for now).
change in magnification
change in focus
• Assume we know f and all D’ values. • How do can we align the images?
𝑚 =𝑓
𝐷′ − 𝑓1
𝐷′+1
𝐷=1
𝑓
resize using these equations
![Page 27: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/27.jpg)
How do you merge a focal stack?1. Align images2. Assign per-pixel weights representing “in-focus”-ness3. Compute image average
![Page 28: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/28.jpg)
Weight assignmentHow do we measure how much “in-focus” each pixel is?
![Page 29: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/29.jpg)
Weight assignmentHow do we measure how much “in-focus” each pixel is?• Measure local sharpness.• This is also how auto-focus works.
How do we measure local sharpness?
![Page 30: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/30.jpg)
Weight assignmentHow do we measure how much “in-focus” each pixel is?• Measure local sharpness.• This is also how auto-focus works.
run Laplacian operator do some Gaussian blurring (why?)
![Page 31: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/31.jpg)
Weight assignmentHow do we measure how much “in-focus” each pixel is?• Measure local sharpness.• This is also how auto-focus works.
run Laplacian operator do some Gaussian blurring (so that nearby pixels have similar weights)
Just one example, many alternatives possible.
![Page 32: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/32.jpg)
How do you merge a focal stack?1. Align images2. Assign per-pixel weights representing “in-focus”-ness3. Compute image average
![Page 33: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/33.jpg)
Focal stack merging
![Page 34: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/34.jpg)
Some results
example image from stack all in-focus image
![Page 35: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/35.jpg)
Another exampleFocal stacking is very useful in macrophotography, where depths of field are very shallow
![Page 36: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/36.jpg)
Another example
middle image from stack all in-focus image
![Page 37: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/37.jpg)
Another look at the mixing weights
What do the mixing weights look like?
![Page 38: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/38.jpg)
Another look at the mixing weights
Depth from defocus = determining sharpest
pixel in focal stack
![Page 39: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/39.jpg)
Depth from defocus on a mobile phone
Use focal stack from autofocus
[Suwajanakorn et al., CVPR 2015]
![Page 40: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/40.jpg)
Can we use both focus and aperture?Confocal stereo
[Hassinof and Kutulakos, ECCV 2006]
![Page 41: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/41.jpg)
Lightfield
![Page 42: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/42.jpg)
Measuring rays
We can capture the same set of rays by using a pinhole camera from multiple viewpoints
![Page 43: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/43.jpg)
Measuring rays
We can capture the same set of rays by using a pinhole camera from multiple viewpoints
![Page 44: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/44.jpg)
Measuring rays
We can capture the same set of rays by using a pinhole camera from multiple viewpoints
![Page 45: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/45.jpg)
Measuring rays
We can capture the same set of rays by using a pinhole camera from multiple viewpoints
![Page 46: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/46.jpg)
Measuring rays
We can capture the same set of rays by using a pinhole camera from multiple viewpoints
![Page 47: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/47.jpg)
Measuring rays
We can capture the same set of rays by using a pinhole camera from multiple viewpoints
![Page 48: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/48.jpg)
Measuring rays
We can capture the same set of rays by using a pinhole camera from multiple viewpoints• How would you merge these images into a lens, defocused image?
![Page 49: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/49.jpg)
Lightfield: all rays in a scene
Parameterize every ray based on its intersections with two planes.
What is the dimension of the lightfield?
aperture planereference plane
![Page 50: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/50.jpg)
Lightfield: all rays in a scene
Parameterize every ray based on its intersections with two planes.
4-dimensional function L(u, v, s, t)
aperture plane (u, v)reference plane (s, t)
![Page 51: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/51.jpg)
Lightfield: all rays in a scene
Parameterize every ray based on its intersections with two planes.
4-dimensional function L(u, v, s, t) (conjugate of scene-based function)
aperture plane (u, v)reference plane (s, t) sensor plane (s, t)
![Page 52: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/52.jpg)
Lightfield slices
4-dimensional function L(u, v, s, t) (conjugate of scene-based function)
aperture plane (u, v) sensor plane (s, t)
What does L(u = uo, v = vo, s, t) look like?
![Page 53: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/53.jpg)
Lightfield slicesaperture plane (u, v) sensor plane (s, t)
What does L(u = uo, v = vo, s, t) look like?• a pinhole image from a certain viewpoint
4-dimensional function L(u, v, s, t) (conjugate of scene-based function)
![Page 54: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/54.jpg)
Lightfield slicesaperture plane (u, v) sensor plane (s, t)
Lightfield slice L(u = uo, v = vo, s, t)
aperture coordinates u = uo, v = vo
reference plane (s, t)
![Page 55: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/55.jpg)
Lightfield slicesaperture plane (u, v) sensor plane (s, t)
What does L(u, v, s = so, t = to) look like?
What does L(u = uo, v = vo, s, t) look like?• a pinhole image from a certain viewpoint
4-dimensional function L(u, v, s, t) (conjugate of scene-based function)
![Page 56: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/56.jpg)
Lightfield slicesaperture plane (u, v) sensor plane (s, t)
What does L(u, v, s = so, t = to) look like?• radiance emitted by a certain (in-focus)
point at various directions
What does L(u = uo, v = vo, s, t) look like?• a pinhole image from a certain viewpoint
4-dimensional function L(u, v, s, t) (conjugate of scene-based function)
![Page 57: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/57.jpg)
Lightfield slicesaperture plane (u, v) sensor plane (s, t)reference plane (s, t)
reference/sensor coordinates s = so, t = to
Lightfield slice L(u, v, s = so, t = to)
reference/sensor coordinates s = so, t = to
![Page 58: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/58.jpg)
Lightfield slicesaperture plane (u, v) sensor plane (s, t)
What does L(u, v, s = so, t = to) look like?• radiance emitted by a certain (in-focus)
point in various directions
What does L(u = uo, v = vo, s, t) look like?• a pinhole image from a certain viewpoint
4-dimensional function L(u, v, s, t) (conjugate of scene-based function)
![Page 59: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/59.jpg)
Lightfield visualization
L(u, v, s = so, t = to) is the radiance emitted by a certain (in-focus)
point at various directions
L(u = uo, v = vo, s, t) is a pinhole image from a certain viewpoint
Demo: http://lightfield.stanford.edu/lfs.html
![Page 60: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/60.jpg)
Lightfield: all rays in a scene
How can you capture the lightfield of a scene?
4-dimensional function L(u, v, s, t) (conjugate of scene-based function)
aperture plane (u, v)reference plane (s, t) sensor plane (s, t)
![Page 61: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/61.jpg)
Measuring lightfields
![Page 62: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/62.jpg)
How to capture a lightfield?
How can you do this?
![Page 63: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/63.jpg)
Option 1: use multiple cameras
(“synthetic aperture”)
Stanford camera array
[Willburn et al., SIGGRAPH 2005]
![Page 64: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/64.jpg)
Option 1: use multiple cameras
Stanford camera array
What kind of lens would you use for this?
[Willburn et al., SIGGRAPH 2005](“synthetic aperture”)
![Page 65: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/65.jpg)
![Page 66: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/66.jpg)
Option 2: take multiple images with one camera
Single camera mounted on LEGO motor. Demo: http://lightfield.stanford.edu/aperture.swf?lightfield=data/self_portrait_lf/preview.zip&zoom=1
How would you move the camera?
![Page 67: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/67.jpg)
Plenoptic camera
![Page 68: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/68.jpg)
Option 3: use a plenoptic camera
plenoptic = plenus (Latin for “full”) + optic (Greek for “seeing”, in this case)
![Page 69: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/69.jpg)
Making a plenoptic cameraaperture plane (u, v) sensor plane (s, t)reference plane (s, t)
reference/sensor coordinates s = so, t = to
Lightfield slice L(u, v, s = so, t = to)
reference/sensor coordinates s = so, t = to
![Page 70: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/70.jpg)
Making a plenoptic cameraaperture plane (u, v) sensor plane (s, t)reference plane (s, t)
Lightfield slice L(u, v, s = so, t = to)
each pixel measures ray
L(u, v, s = so, t = to) for different (u, v)
reference/sensor coordinates s = so, t = to
reference/sensor coordinates s = so, t = to
![Page 71: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/71.jpg)
Making a plenoptic cameraaperture plane (u, v) sensor plane (s, t)reference plane (s, t)
Lightfield L(u, v, s, t)
each pixel measures a ray
L(u, v, s, t)
each pinhole corresponds to a slice
L(u, v, s = so, t = to)
How can we make this more light efficient?
![Page 72: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/72.jpg)
Making a plenoptic cameraaperture plane (u, v) sensor plane (s, t)reference plane (s, t)
Lightfield L(u, v, s, t)
each pixel measures a ray
L(u, v, s, t)
each pinhole corresponds to a slice
L(u, v, s = so, t = to)
How can we make this more light efficient?• replace pinholes with lenslets
![Page 73: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/73.jpg)
Prototype plenoptic camera
[Ng et al., Stanford Technical Report 2005]
![Page 74: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/74.jpg)
Commercial plenoptic camera
![Page 75: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/75.jpg)
Commercial plenoptic camera
lenslet array
![Page 76: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/76.jpg)
Commercial plenoptic camera
newer version with higher resolution
![Page 77: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/77.jpg)
Making a plenoptic camera
Is it possible to make a plenoptic camera using an array of pinholes?
![Page 78: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/78.jpg)
Making a plenoptic camera
Is it possible to make a plenoptic camera using an array of pinholes?• Yes, under certain conditions on the images being captured.
Need to be able to separate different rays falling on same pixel.
[Georgiev, ECCV 2008]
![Page 79: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/79.jpg)
Images from lightfields
![Page 80: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/80.jpg)
A plenoptic “image”
What are these circles?
![Page 81: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/81.jpg)
A plenoptic cameraaperture plane (u, v) sensor plane (s, t)reference plane (s, t)
Lightfield L(u, v, s, t)
each pinhole corresponds to a slice
L(u, v, s = so, t = to)
![Page 82: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/82.jpg)
How do I…
Simulate different viewpoints?
![Page 83: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/83.jpg)
How do I…
Simulate different viewpoints?• Pick same pixel within each aperture view
![Page 84: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/84.jpg)
How do I…
Simulate different viewpoints?• Pick same pixel within each aperture view
Simulate different aperture sizes?
![Page 85: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/85.jpg)
How do I…
Simulate different viewpoints?• Pick same pixel within each aperture view
Simulate different aperture sizes?• Sum more than one pixels within each
aperture view
![Page 86: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/86.jpg)
How do I…
Simulate different viewpoints?• Pick same pixel within each aperture view
Simulate different aperture sizes?• Sum more than one pixels within each
aperture view
Simulate different focus depths?
![Page 87: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/87.jpg)
How do I…
Simulate different viewpoints?• Pick same pixel within each aperture view
Simulate different aperture sizes?• Sum more than one pixels within each
aperture view
Simulate different focus depths?• Shift the areas you integrate relative
to each other
![Page 88: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/88.jpg)
![Page 89: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/89.jpg)
![Page 90: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/90.jpg)
![Page 91: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/91.jpg)
![Page 92: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/92.jpg)
[Ng, SIGGRAPH 2006]
![Page 93: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/93.jpg)
Refocusing example
![Page 94: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/94.jpg)
Refocusing example
![Page 95: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/95.jpg)
Refocusing example
![Page 96: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/96.jpg)
Refocusing example
Many more examples with demo: http://lightfields.stanford.edu/
![Page 97: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/97.jpg)
Three ways to measure a lightfield
1) Use a plenoptic camera 2) Use a camera array3) Use one camera
multiple times
What are the pros and cons of each?
![Page 98: Focal stacks and lightfieldsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 24. · lens-object distance D lens-sensor ... Which of these parameters would you](https://reader036.vdocument.in/reader036/viewer/2022071410/610554042b2e1a2986134805/html5/thumbnails/98.jpg)
ReferencesBasic reading:• Szeliski textbook, Section 12.1.3, 13.3.
Additional reading:• Nayar et al., “Real-Time Focus Range Sensor,” PAMI 1996.
one of the (relatively) early papers on depth from defocus.• Suwajanakorn et al., “Depth from Focus with Your Mobile Phone,” CVPR 2015.
implementing depth from defocus on a mobile phone using the autofocus focal stack.• Hasinoff and Kutulakos, “Confocal Stereo,” IJCV 2009.
the paper on high resolution depth from a focus and aperture stack.• Kuthirummal et al., “Flexible Depth of Field Photography,” PAMI 2010.
continuously change focus within one exposure, without stopping to capture a stack.• Levoy and Hanrahan, “Light Field Rendering,” SIGGRAPH 1996. • Gortler et al., “The Lumigraph,” SIGGRAPH 1996.
the two papers introducing the light field. • Adelson and Wang, “Single Lens Stereo with a Plenoptic Camera,” PAMI 1992.
the paper (re)-introducing the plenoptic camera to computer vision and graphics.• Ng et al., “Light field photography with a hand-held plenoptic camera,” Stanford TR 2005.
the paper (re)-(re)-introducing the plenoptic camera, and the precursor to Lytro.• Ng, “Fourier Slice Photography,” SIGGRAPH 2005.
the paper on frequency-space analysis of refocusing and lightfield measurements.• Wilburn et al., “High Performance Imaging Using Large Camera Arrays,” SIGGRAPH 2005.
the camera array paper.• Georgiev et al., “Unified Frequency Domain Analysis of Lightfield Cameras,” ECCV 2008.
make a lightfield camera from a pinhole array (and many other interesting stuff about lightfield cameras).