neural rendering - mit deep learning...
TRANSCRIPT
![Page 1: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/1.jpg)
Neural Rendering
Chuan Li
Lambda Labs
Collaborators: Thu Nguyen-Phuoc, Bing Xu, Yongliang Yang, Stephen Balaban, Lucas Theis, Christian Richardt, Junfei Zhang, Rui Wang, Kun Xu, Rui Tang
![Page 2: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/2.jpg)
Model Pictures
Forward (Computer Graphics)
![Page 3: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/3.jpg)
Model Pictures
Forward (Computer Graphics)
Inverse (Computer Vision)
![Page 4: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/4.jpg)
![Page 5: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/5.jpg)
![Page 6: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/6.jpg)
Integral of the incident radians
![Page 7: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/7.jpg)
BRDF
![Page 8: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/8.jpg)
![Page 9: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/9.jpg)
![Page 10: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/10.jpg)
32K SPP Ray Tracing (90 mins 12 CPU Cores)The Tungsten Renderer
![Page 11: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/11.jpg)
P0
P1
![Page 12: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/12.jpg)
P0
P1
![Page 13: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/13.jpg)
R01 | T
01
P0
P1
![Page 14: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/14.jpg)
Inverse (Computer Vision)
R01 | T
01
P0
P1
![Page 15: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/15.jpg)
P2
R12 | T12
Inverse (Computer Vision)
R01 | T
01
P0
P1
![Page 16: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/16.jpg)
Building Rome in a DaySameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz and Richard Szeliski
![Page 17: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/17.jpg)
Model Pictures
Sub-module
End-2-End
Differentiable Rendering
![Page 18: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/18.jpg)
1 SPP
2048 SPP
![Page 19: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/19.jpg)
Sub-modules
Mastering the game of Go with deep neural networks and tree searchDavid Silver et al.
![Page 20: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/20.jpg)
Sub-modules
Value Network
Mastering the game of Go with deep neural networks and tree searchDavid Silver et al.
![Page 21: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/21.jpg)
Sub-modules
Value Network
Policy Network
Mastering the game of Go with deep neural networks and tree searchDavid Silver et al.
![Page 22: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/22.jpg)
2^15 SPP4 SPP
Value Networks
Denoising
![Page 23: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/23.jpg)
2^15 SPP
Value Networks
Denoising
Policy Networks
Same SPP
4 SPP
![Page 24: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/24.jpg)
2^15 SPP
Value Networks
Denoising
Policy Networks
Same SPP
4 SPP
![Page 25: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/25.jpg)
4 SPP Denoised1 sec 2080 Ti
32K SPP Ray Tracing90 mins 12 cores CPU
Adversarial Monte Carlo denoising with conditioned auxiliary feature modulationB Xu et al. Siggraph Asia 2019
![Page 26: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/26.jpg)
loss
DecoderEncoderInput x
Ref
Output
Adversarial Monte Carlo denoising with conditioned auxiliary feature modulationB Xu et al. Siggraph Asia 2019
![Page 27: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/27.jpg)
L1 VGG Loss
L1 VGG Loss + GAN
Adversarial Monte Carlo denoising with conditioned auxiliary feature modulationB Xu et al. Siggraph Asia 2019
![Page 28: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/28.jpg)
loss
DecoderEncoderDiffuseInput x Diffuse
Output
DecoderEncoderSpecular
Input x SpecularOutput Ref
Output
Adversarial Monte Carlo denoising with conditioned auxiliary feature modulationB Xu et al. Siggraph Asia 2019
![Page 29: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/29.jpg)
Auxiliary
loss
DecoderEncoderDiffuseInput x Diffuse
Output
DecoderEncoderSpecular
Input x SpecularOutput Ref
Output
Albedo, normal, depth
![Page 30: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/30.jpg)
Auxiliary
Conv
LeakyReLU
Conv
x
Element-wiseBiasing
![Page 31: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/31.jpg)
Conv
Auxiliary
LeakyReLU
Conv
Conv
LeakyReLU
Conv
x
Element-wiseBiasing
Element-wiseScaling
![Page 32: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/32.jpg)
Conv
Auxiliary
LeakyReLU
Conv
Conv
LeakyReLU
Conv
x
Element-wiseBiasing (OR)
Element-wiseScaling (AND)
![Page 33: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/33.jpg)
Denoise comparison 4 SPP
Adversarial Monte Carlo denoising with conditioned auxiliary feature modulationB Xu et al. Siggraph Asia 2019
![Page 34: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/34.jpg)
2^15 SPP
Value Networks
Denoising
Policy Networks
Same SPP
4 SPP
![Page 35: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/35.jpg)
Neural Importance SamplingThomas Müller et al. ACM Transactions on Graphics 2019
![Page 36: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/36.jpg)
incidence radiance map
Neural Importance SamplingThomas Müller et al. ACM Transactions on Graphics 2019
![Page 37: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/37.jpg)
![Page 38: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/38.jpg)
Neural Importance SamplingThomas Müller et al. ACM Transactions on Graphics 2019
![Page 39: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/39.jpg)
Neural Importance SamplingThomas Müller et al. ACM Transactions on Graphics 2019
![Page 40: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/40.jpg)
Model Pictures
Sub-module
End-2-End
Differentiable Rendering
![Page 41: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/41.jpg)
Ray TracingImage Centric
RasterizationObject Centric
![Page 42: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/42.jpg)
Ray TracingImage Centric
RasterizationObject Centric
Visibility
![Page 43: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/43.jpg)
Ray TracingImage Centric
RasterizationObject Centric
Shading
![Page 44: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/44.jpg)
Depth Map Voxel Point Cloud Mesh
Memory Good Very Poor Poor Very Good
NN friendly Great Yes No Enemy
![Page 45: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/45.jpg)
Depth Map Voxel Point Cloud Mesh
Memory Good Very Poor Poor Very Good
NN friendly Great Yes No Enemy
![Page 46: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/46.jpg)
Depth Map Voxel Point Cloud Mesh
Memory Good Very Poor Poor Very Good
NN friendly Great Yes No Enemy
![Page 47: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/47.jpg)
Depth Map Voxel Point Cloud Mesh
Memory Good Very Poor Poor Very Good
NN friendly Great Yes No Enemy
![Page 48: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/48.jpg)
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
![Page 49: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/49.jpg)
Neural Voxels
32 x 32 x 32 x 16
3D Encoder
Neural Voxels
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
![Page 50: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/50.jpg)
Neural Voxels
32 x 32 x 32 x 16
3D Encoder
3D-2D
32 x 32 x 512
Neural Pixels
VisibilityNeural Voxels
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
![Page 51: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/51.jpg)
Neural Voxels
32 x 32 x 32 x 16
3D Encoder
3D-2D
32 x 32 x 512
Neural Pixels
VisibilityNeural Voxels
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
![Page 52: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/52.jpg)
Neural Voxels
32 x 32 x 32 x 16
3D Encoder
3D-2D
32 x 32 x 512
Neural Pixels
2D Decoder
ShadingNeural Voxels Visibility
MSE pixel loss
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
![Page 53: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/53.jpg)
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
![Page 54: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/54.jpg)
Contour
Toon
Ambient OcclusionRenderNet: A deep convolutional network for differentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018
![Page 55: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/55.jpg)
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
![Page 56: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/56.jpg)
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
![Page 57: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/57.jpg)
3D Encoder
3D-2D Neural Pixels
2D Decoder
TextureNetwork
NeuralTexture Voxels
or
Neural Voxels
Channel-wise Concatenation
64 x 64 x 64 x 4
64 x 64 x 64 x 1
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
![Page 58: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/58.jpg)
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
![Page 59: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/59.jpg)
Same shape, different textures
Same texture, different shapes
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
![Page 60: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/60.jpg)
Depth Map Voxel Point Cloud Mesh
Memory Good Very Poor Poor Very Good
NN friendly Great Yes No Enemy
![Page 61: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/61.jpg)
Rasterization a RGB point cloud
Neural Point-Based GraphicsKA Aliev et al, arxiv 2019
![Page 62: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/62.jpg)
Rasterization a neural point cloud
(First three PCA dimensions of the neural descriptor)
Neural Point-Based GraphicsKA Aliev et al, arxiv 2019
![Page 63: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/63.jpg)
Rasterization a neural point cloud
(First three PCA dimensions of the neural descriptor)
Neural Point-Based GraphicsKA Aliev et al, arxiv 2019
![Page 64: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/64.jpg)
Neural Point-Based GraphicsKA Aliev et al, arxiv 2019
RB
G ra
ster
izat
ion
Neu
ral r
aste
rizat
ion
![Page 65: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/65.jpg)
Neural 3D Mesh RendererH Kato et al, CVPR 2018
Deferred Neural Rendering: Image Synthesis using Neural Textures
J Thies et al, Siggraph 2019
![Page 66: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/66.jpg)
Model Pictures
Sub-module
End-2-End
![Page 67: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/67.jpg)
?
![Page 68: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/68.jpg)
TargetApproximation
![Page 69: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/69.jpg)
TargetApproximation RenderedApproximation
![Page 70: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/70.jpg)
Loss
Back-propagate
TargetRenderedApproximation
Approximation
![Page 71: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/71.jpg)
Loss
TargetRenderedApproximation
UpdatedApproximation
Back-propagate
![Page 72: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/72.jpg)
Loss
TargetRenderedApproximation
UpdatedApproximation
Back-propagateFor Free
![Page 73: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/73.jpg)
Loss
TargetRenderedApproximation
UpdatedApproximation
Back-propagate
Expensive
![Page 74: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/74.jpg)
Loss
TargetRenderedApproximation
DecoderEncoder
Reconstruction Rendering
![Page 75: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/75.jpg)
Human perception imposes coordinate frame on objects
Inductive Bias: Separate Appearance from Pose
![Page 76: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/76.jpg)
Learning 3D representation from natural images without 3D supervision
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
![Page 77: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/77.jpg)
Conditional GANs
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
![Page 78: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/78.jpg)
Info GANs
Conditional GANs
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
![Page 79: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/79.jpg)
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
![Page 80: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/80.jpg)
RenderNet3D Generator
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
![Page 81: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/81.jpg)
RenderNet3D Generator
3D StyleGAN
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
![Page 82: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/82.jpg)
RenderNet3D Generator
3D StyleGAN
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
![Page 83: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/83.jpg)
RenderNet3D Generator
3D StyleGAN
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
![Page 84: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/84.jpg)
RenderNet3D Generator
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
![Page 85: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/85.jpg)
RenderNet3D Generator
Real/Fake
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
![Page 86: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/86.jpg)
RenderNet3D Generator
A representation that is unbreakable under 3D rigid-body transformations
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
![Page 87: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/87.jpg)
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
![Page 88: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/88.jpg)
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
![Page 89: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/89.jpg)
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
![Page 90: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/90.jpg)
z1 z2
Shape Controller Texture Controller
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
![Page 91: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/91.jpg)
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
![Page 92: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/92.jpg)
Model Pictures
Forward (Computer Graphics)
Inverse (Computer Vision)
![Page 93: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/93.jpg)
Model Pictures
Sub-module for Ray Tracing (Value / Policy Networks)
End-2-End Rasterization (Depthmap, Voxel, Point Cloud, Mesh)
Differentiable Rendering (Representation Learning)
![Page 94: Neural Rendering - MIT Deep Learning 6.S191introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L9.pdf · Neural Voxels 32 x 32 x 32 x 16 3D Encoder 3D-2D 32 x 32 x 512 Neural Pixels](https://reader036.vdocument.in/reader036/viewer/2022062505/5edad06109ac2c67fa6858e2/html5/thumbnails/94.jpg)
Thu Nguyen-Phuoc Bing Xu Yongliang Yang Stephen Balaban
Lucas Theis Christian Richardt Junfei Zhang Rui Wang Kun Xu Rui Tang