advanced computer graphics (spring 2005) coms 4162, lecture 21: image-based rendering ravi...
Post on 20-Dec-2015
223 views
TRANSCRIPT
![Page 1: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/1.jpg)
Advanced Computer Graphics Advanced Computer Graphics (Spring 2005) (Spring 2005)
COMS 4162, Lecture 21: Image-Based Rendering
Ravi Ramamoorthi
http://www.cs.columbia.edu/~cs4162
![Page 2: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/2.jpg)
To Do / MotivationTo Do / Motivation
Work hard on assignment 4
This last series of lectures covers (at a high level) some more advanced topics and areas of current research interest in modern rendering
![Page 3: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/3.jpg)
Course OutlineCourse Outline
3D Graphics Pipeline
Rendering(Creating, shading images from geometry, lighting, materials)
Modeling(Creating 3D Geometry)
![Page 4: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/4.jpg)
Next few slides courtesy Paul Debevec; SIGGRAPH 99 course notes
![Page 5: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/5.jpg)
![Page 6: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/6.jpg)
![Page 7: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/7.jpg)
Image-Based Modeling and RenderingImage-Based Modeling and Rendering
Generate new views of a scene directly from existing views
“Pure” IBR (such as lightfields): no geometric model of scene
Other IBR techniques try to obtain higher quality with less storage by building a model
![Page 8: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/8.jpg)
IBR: Pros and ConsIBR: Pros and Cons
Advantages Easy to capture images: photorealistic by definition Simple, universal representation Often bypass geometry estimation? Independent of scene complexity?
Disadvantages WYSIWYG but also WYSIAYG Explosion of data as flexibility increased Often discards intrinsic structure of model?
![Page 9: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/9.jpg)
![Page 10: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/10.jpg)
![Page 11: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/11.jpg)
IBR: A brief historyIBR: A brief history
Texture maps, bump maps, env. maps [70s]
Poggio et al. MIT: Faces, image-based analysis/synthesis
Modern Era Chen and Williams 93, View Interpolation [Images with depth] Chen 95 Quicktime VR [Images from many viewpoints] McMillan and Bishop 95 Plenoptic Modeling [Images w disparity] Gortler et al, Levoy and Hanrahan 96 Light Fields [4D] Shade et al. 98 Layered Depth Images [2.5D] Debevec et al. 00 Reflectance Field [4D] Inverse rendering methods (Sato,Yu,Marschner,Boivin,…)
Fundamentally, sampled representations in graphics
![Page 12: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/12.jpg)
OutlineOutline
Overview of IBR
Basic approaches Image Warping Light Fields Survey of some recent work
![Page 13: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/13.jpg)
Warping slides courtesy Leonard McMillan
![Page 14: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/14.jpg)
![Page 15: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/15.jpg)
![Page 16: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/16.jpg)
![Page 17: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/17.jpg)
![Page 18: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/18.jpg)
OutlineOutline
Overview of IBR
Basic approaches Image Warping
[2D + depth. Requires correspondence/disparity] Light Fields [4D] Survey of some recent work
![Page 19: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/19.jpg)
![Page 20: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/20.jpg)
Plenoptic FunctionPlenoptic Function
L(x,y,z,,,t,)
Captures all light flow in a scene to/from any point (x,y,z), in any direction (,), at any time (t), at any frequency ()
Enough information toconstruct any imageof the scene at any time
(x,y,z)(x,y,z)((,,))
[Funkhouser][Funkhouser]
![Page 21: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/21.jpg)
Plenoptic Function SimplificationsPlenoptic Function Simplifications
Represent color as RGB: eliminate
Static scenes: ignore dependence on t
7D 3 5D
![Page 22: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/22.jpg)
![Page 23: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/23.jpg)
![Page 24: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/24.jpg)
![Page 25: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/25.jpg)
![Page 26: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/26.jpg)
![Page 27: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/27.jpg)
![Page 28: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/28.jpg)
![Page 29: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/29.jpg)
![Page 30: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/30.jpg)
![Page 31: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/31.jpg)
Lumigraph PostprocessingLumigraph Postprocessing
Obtain rough geometric model Chroma keying (blue screen) to extract silhouettes Octree-based space carving
Resample images to two-plane parameterization
![Page 32: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/32.jpg)
Lumigraph RenderingLumigraph Rendering
Use rough depth information to improve rendering quality
![Page 33: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/33.jpg)
Lumigraph RenderingLumigraph Rendering
Use rough depth information to improve rendering quality
![Page 34: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/34.jpg)
Lumigraph RenderingLumigraph Rendering
Without usinggeometry
Using approximategeometry
![Page 35: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/35.jpg)
Unstructured Lumigraph RenderingUnstructured Lumigraph Rendering
Further enhancement of lumigraphs:do not use two-plane parameterization
Store original pictures: no resampling
Hand-held camera, moved around an environment
![Page 36: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/36.jpg)
Unstructured Lumigraph RenderingUnstructured Lumigraph Rendering
To reconstruct views, assign penalty to each original ray Distance to desired ray, using
approximate geometry Resolution Feather near edges of image
Construct “camera blending field”
Render using texture mapping
![Page 37: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/37.jpg)
Unstructured Lumigraph Unstructured Lumigraph RenderingRendering
Blending field Rendering
![Page 38: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/38.jpg)
OutlineOutline
Overview of IBR
Basic approaches Image Warping
[2D + depth. Requires correspondence/disparity] Light Fields [4D] Survey of some recent work
![Page 39: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/39.jpg)
LDIsLDIs
Layered depth images [Shade et al. 98]
Geometry
Camera
Slide from Agrawala, Ramamoorthi, Heirich, Moll, SIGGRAPH 2000
![Page 40: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/40.jpg)
LDIsLDIs
Layered depth images [Shade et al. 98]
LDI
![Page 41: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/41.jpg)
LDIsLDIs
Layered depth images [Shade et al. 98]
LDI
(Depth, Color)
![Page 42: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/42.jpg)
![Page 43: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/43.jpg)
Surface Light FieldsSurface Light Fields
Miller 98, Nishino 99, Wood 00
Reflected light field (lumisphere) on surface
Explicit geometry as against light fields. Easier compress
![Page 44: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/44.jpg)
Acquiring Reflectance Field of Human Acquiring Reflectance Field of Human Face [Debevec et al. SIGGRAPH 00]Face [Debevec et al. SIGGRAPH 00]
Illuminate subject from many incident directions
![Page 45: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/45.jpg)
Example ImagesExample Images
Images from Debevec et al. 00
![Page 46: Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi cs4162](https://reader036.vdocument.in/reader036/viewer/2022062320/56649d425503460f94a1d569/html5/thumbnails/46.jpg)
Conclusion (my views)Conclusion (my views)
Real issue is compactness/flexibility vs. rendering speed
IBR is use of sampled representations. Easy to interpolate, fast to render. If samples images, easy to acquire.
Of course, for this course, some pretty fancy precomputed algorithms (because we want to handle complex lighting that changes)
IBR in pure form not really practical WYSIAYG Explosion as increase dimensions (8D transfer function) Ultimately, compression, flexibility needs geometry/materials But lots of recent work (some in course) begins to correct these issues
Right question is tradeoff compactness/efficiency Factored representations Understand sampling rates and reconstruction