![Page 1: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/1.jpg)
Reusing Pixels for Interactive Global
Illumination
Reusing Pixels for Interactive Global
Illumination
Bruce WalterCornell University
![Page 2: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/2.jpg)
Interactive Global IlluminationInteractive Global Illumination
• Problem: global illumination methods are too slow for interactive use
Ray tracing Path tracing
![Page 3: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/3.jpg)
Interactive Global IlluminationInteractive Global Illumination
• Do we need to render every pixel of every frame?
Frame 10 Frame 20 Frame 30
![Page 4: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/4.jpg)
Display ProcessDisplay Process
• Automatically exploit spatial and temporal coherence
• Layered on top of an existing (slow) global illumination renderer
• Provide interactive performance
![Page 5: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/5.jpg)
Interactive Global IlluminationInteractive Global Illumination
renderer
application
image
user
• Standard visual feedback loop–Entirely
synchronous–Framerate is
limited by the renderer
![Page 6: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/6.jpg)
Interactive Global IlluminationInteractive Global Illumination
• Modified visual feedback loop
Display
Application
Image
UserRenderer
Asynchronousinterface
![Page 7: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/7.jpg)
Interactive Global IlluminationInteractive Global Illumination
• Interactive requirements– Image quality–Responsiveness
• Don’t make the user wait• Provide rapid user feedback
–Consistency• Don’t surprise or distract the user• Avoid sudden changes if possible
– Eg, in quality, frame rate, popping, etc.
![Page 8: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/8.jpg)
IssuesIssues
• How to cache results for reuse?– Interpolate images from sparse data–Handle camera motion between frames
• Handle occlusion changes
• Which samples should be rendered?–Prioritize for maximum benefit
• What if scene or shading changes?–Detect and discard data that is no
longer valid
![Page 9: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/9.jpg)
ApproachesApproaches
• Image based–Post-rendering Warp (Mark97)–Corrective texturing (Stamminger00)
• Point based–Render Cache (Walter99,02)
• Mesh based–Holodeck (Ward98,99)–Tapestry (Simmons00)–Shading Cache (Tole02)
![Page 10: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/10.jpg)
Post-Rendering 3D WarpPost-Rendering 3D Warp
• Render subset of frames–E.g, every 6th frame is rendered
• Use standard image warping techniques to compute the other frames
![Page 11: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/11.jpg)
Post-Rendering 3D WarpPost-Rendering 3D Warp
• Problem: holes and missing data
Reference frame Warped frame
The camera is moving to the left in this example.
![Page 12: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/12.jpg)
Post-Rendering 3D WarpPost-Rendering 3D Warp
• Warp from both past and future reference frames–Heuristics for combining pixel results
Prior reference Warped frame Next reference
![Page 13: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/13.jpg)
Post-Rendering 3D WarpPost-Rendering 3D Warp
• Must predict the locations of future frames–Longer predictions become rapidly less
accurate
Camera path
Warped frames
Predicted frames
![Page 14: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/14.jpg)
Corrective TexturingCorrective Texturing
• Start with a standard hardware rendering of scene–Graphics hardware very good at
interactive display–Start with a radiosity solution
• Compare to underlying renderer–Apply corrections where they differ–Corrections applied as projective
textures
![Page 15: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/15.jpg)
Corrective TexturingCorrective Texturing
• Corrective textures are dynamically assigned to objects
![Page 16: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/16.jpg)
Corrective TexturingCorrective Texturing
Radiositysolution
Correctedimage
Correctivetexture
![Page 17: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/17.jpg)
Corrective TexturingCorrective Texturing
• Sparse rendered samples compared to hardware displayed results–Differences splatted into textures–More samples generated near points
which had large differences–Samples which are likely to have
changed are deprecated so that can be overwritten by future results
![Page 18: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/18.jpg)
Render CacheRender Cache
• Results stored as cloud of unordered points with:–3D position (located on surfaces)–Color–Age –Object identifier
![Page 19: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/19.jpg)
Render CacheRender Cache
• Reproject points into current frame–Occlusion errors–Holes in data
Initial view After reprojection
![Page 20: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/20.jpg)
Render CacheRender Cache
• Use occlusion culling heuristic• Interpolation to fill holes
–Fixed size kernels, 3x3 and 7x7
Reprojection Occlusion cull Interpolation
![Page 21: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/21.jpg)
Render CacheRender Cache
• Priority image for sampling–High priority for sparse regions–High priority for old points
• Convert priority image to sparse set of locations to be rendered–Uses error-diffusion dither
• Also uses predictive sampling–Try to sample new regions just before
they become visible
![Page 22: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/22.jpg)
Render CacheRender Cache
Displayed image Priority image Requested pixels
![Page 23: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/23.jpg)
Render CacheRender Cache
• Recomputes old samples to detect changes–Nearby points are aged to raise priority
and cause point invalidation
• Object motion–Associated points can be transformed
along with the object
![Page 24: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/24.jpg)
Point & Edge RenderingPoint & Edge Rendering
Edges Edges and samples Output image
• Tracks and uses discontinuity edges• SIGGRAPH 2003 paper
–Presentation: Wednesday 1:45pm
![Page 25: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/25.jpg)
HolodeckHolodeck
• Uses Radiance as its renderer• Rendered samples stored in a 4D
data structure–Similar to Light Field or Lumigraph–Can be very large
• Paged to disk if necessary
– Lazily evaluated• Samples generated near current viewpoint
–Position and other parameters are specified by the user
![Page 26: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/26.jpg)
HolodeckHolodeck
• Uses Gouraud-shaded triangle mesh–Get samples near current viewpoint–Samples become vertices in a mesh–Delaunay triangulation of samples in
direction space about a center of projection
–Hardware provides fast display including interpolation between samples
![Page 27: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/27.jpg)
HolodeckHolodeck
• Mesh construction–Choose center point–Construct Delaunay triangulation
• Based on sample point’s projection onto a sphere about the center point
–Display mesh using hardware–Update incrementally with new samples
• If user moves too far, then must choose new center and rebuild mesh
![Page 28: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/28.jpg)
HolodeckHolodeck
• Depth heuristic to reduce occlusion errors
• Special techniques for designated moving objects
![Page 29: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/29.jpg)
TapestryTapestry
• Based on Holodeck system with several enhancements:–Prioritized sampling– Incremental “recentering” of spherical
Delaunay mesh as viewpoint moves–Fixed cache size
• Max vertices = pixels
–Sample invalidation• Occlusion and color change heuristics
![Page 30: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/30.jpg)
TapestryTapestry
• Each triangle assigned a priority–Color & depth differences and age–Rasterize priority using hardware–Quasi-random sampling with rejection
Image Priority Samples
![Page 31: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/31.jpg)
Shading CacheShading Cache
• Display mesh is refinement of original scene mesh–No occlusion errors–Hardware handles textures–Display mesh >= original mesh–Easier to handle moving objects
• Decouples frame update from mesh update
![Page 32: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/32.jpg)
Shading CacheShading Cache
• Adds flood-fill heuristic for sampling–Discontinuities require locally dense
subdivision
• Mesh de-refinement– If not recently visible– If denser than pixel spacing– If color changes are detected
![Page 33: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/33.jpg)
Shading CacheShading Cache
![Page 34: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/34.jpg)
ComparisonComparison
Warp
Corrective Tex.
Render Cache
Holodeck
Tapestry
Shading Cache
Typicalframe rates
20 - 60 fps
5 - 10 fps
10 - 20 fps
NA
3 - 10 fps
30 - 60 fps
Sparseness ratio
4 - 10
250 - 1000
8 - 100
NA
2000 - 8000
1000 - 20000
Target rendererspeed
< 1s
20 - 200s
.5 - 10s
200 - 1000s
500 - 1000s
50 - 1000s
![Page 35: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/35.jpg)
ComparisonComparison
Warp
Corrective Tex.
Render Cache
Holodeck
Tapestry
Shading Cache
Moving objects
No
No
Limited
Special
No
Yes
Independent ofscene complexity
Yes
No
Yes
No
Yes
No
Hardwareaccelerated
No
Yes
No
Yes
Yes
Yes
![Page 36: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/36.jpg)
Downloadable VersionsDownloadable Versions
• Render Cache–http://www.graphics.cornell.edu/researc
h/interactive/rendercache/
• Holodeck–http://radsite.lbl.gov/radiance
![Page 37: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/37.jpg)
BibliographyBibliography
• W. Mark, L. McMillan, and G. Bishop, “Post-Rendering Warping”, Proceedings of 1997 Symposium on Interactive 3D Graphics, 1997.– http://www.cs.utexas.edu/users/billmark/i3dwww/i3d.html
• M. Stamminger, J. Haber, H. Schirmacher, H.-P. Seidel, “Walkthroughs with Corrective Texturing”, Rendering Techniques 2000 (Proc. EurographicsWorkshop on Rendering), 2000– http://www-sop.inria.fr/reves/personnel/Marc.Stamminger/corrtex/
![Page 38: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/38.jpg)
BibliographyBibliography
• B. Walter, G. Drettakis, S. Parker, “Interactive Rendering Using the Render Cache”, Rendering Techniques '99, (Proc. 10th Eurographics Workshop on Rendering), 1999
• B. Walter, G. Drettakis, D. P. Greenberg, “Enhancing and Optimizing the Render Cache”, Rendering Techniques ‘02, (Proc. 13th Eurographics Workshop on Rendering), 2002– http://www.graphics.cornell.edu/research/interactive/rendercache/
![Page 39: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/39.jpg)
BibliographyBibliography
• G. W. Larson, “The Holodeck: A Parallel Ray-caching Rendering System'’, Proceedings of the Second EurographicsWorkshop on Parallel Graphics andVisualisation, September 1998.
• G. Ward, and M. Simmons, “The HolodeckRay Cache: An Interactive Rendering System for Global Illumination inNondiffuse Environments,'' ACM Transactions on Graphics, 18(4):361-98, October 1999.
![Page 40: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/40.jpg)
BibliographyBibliography
• M. Simmons and C. H. SÈquin "Tapestry: A Dynamic Mesh-based Display Representation for Interactive Rendering”, Rendering Techniques 2000, (Proc. of the 11th Eurographics Workshop on Rendering), 2000– http://www.cs.berkeley.edu/~simmons/
• P. Tole, F. Pellacini, B. Walter, and D. P. Greenberg, ”Interactive global illumination in dynamic scenes”, SIGGRAPH 2002 Conference Proceedings, 2002.
![Page 41: Reusing Pixels for Interactive Global Illuminationbjw/IDPCourse/Walter.display.course.pdf · Reference frame Warped frame The camera is moving to the left in this example. Post-Rendering](https://reader033.vdocument.in/reader033/viewer/2022043006/5f902d1db675dd6dec03d801/html5/thumbnails/41.jpg)
UpdatesUpdates
• For the latest version of this talk, go to:– http://www.graphics.cornell.edu/~bjw/IDPCourse/