geometry-aware neural rendering - josh tobinjosh-tobin.com/assets/pdf/geometry_aware_neural... ·...
TRANSCRIPT
Geometry-Aware Neural Rendering
Josh Tobin, OpenAI Robotics, Pieter Abbeel
Josh Tobin Geometry-Aware Neural Rendering
How to model complex robots scenes?
!2
From… …To
Josh Tobin Geometry-Aware Neural Rendering
Model the state of all objects?
!3
• Scales with scene complexity
• How to deal with complex internal state?
• How to deal with out-of-distribution?
Josh Tobin Geometry-Aware Neural Rendering
Only use the state implicitly?
!4
• Can be data inefficient
• May require learning from scratch (which can be dangerous)
• Often lacks reusability
Josh Tobin Geometry-Aware Neural Rendering
Model the 3D structure of the scene?
!5
• High-dimensional representations - scale poorly with scene detail
• No notion of semantics
Josh Tobin Geometry-Aware Neural Rendering
Model the 3D structure implicitly?
!6
Random viewpoints Model
Arbitrary “query”
viewpoint
The Neural Rendering problem
Josh Tobin Geometry-Aware Neural Rendering
Motivation: Generative Query Nets
!7
Eslami, SM Ali, et al. "Neural scene representation and rendering." Science 360.6394 (2018): 1204-1210.
Josh Tobin Geometry-Aware Neural Rendering
Motivation: Generative Query Nets
!8
Eslami, SM Ali, et al. "Neural scene representation and rendering." Science 360.6394 (2018): 1204-1210.
Josh Tobin Geometry-Aware Neural Rendering
Key questions
!9
• Can it scale to high(er)-dimensional images (GQN is 64x64)?
• Does it work for objects with complex state?
• Does it work for a wide range of realistic objects?
• Is it useful for downstream robotics tasks?
Josh Tobin Geometry-Aware Neural Rendering
Potential limitations
!10
• Scene representation is a sum — each feature contains only local information
• Rendering process cannot interact with the full representations (except through backprop)
Josh Tobin Geometry-Aware Neural Rendering
Background: Epipolar Geometry
!11
Josh Tobin Geometry-Aware Neural Rendering
Our Approach
!12
attn attnattnattn attn attnepi
r1
v1
f
e1epi
r1
v1
f
e1
vq
epi
r1
v1
f
e1
attn
clstm h 1h 0 h L
z1
attn
⋯ clstm
zL
attn ⋯
Our ModelPrevious Method Ground Truth
r
1
23
4Representation network
Attention mechanism
Generation network
loss
clstm
z2
Josh Tobin Geometry-Aware Neural Rendering
Epipolar extraction
!13
hl rk
ek
vqvk
Josh Tobin Geometry-Aware Neural Rendering
Attention mechanism
!14
akl
h l−1
1 × 1 × 1 1 × 11 × 1 × 1
ek
h ′� × w′� × d′�h ′� × w′� × h ′� × d′�
h ′� × w′� × h ′� × dv h ′� × w′� × h ′� × dk h ′� × w′� × dk
h ′�w′� × h ′� × dk h ′�w′� × dk × 1
softmax
h ′�w′� × 1 × h ′�h ′�w′� × h ′� × dv
h ′�w′� × dv
h ′�w′� × h ′�
h ′� × w′� × dv
1 × 1
akl
QlKk
Vk
Josh Tobin Geometry-Aware Neural Rendering
Does it help?
!15
Josh Tobin Geometry-Aware Neural Rendering
Examples
!16
Context TargetGQN E-GQN
Josh Tobin Geometry-Aware Neural Rendering
Examples
!17
Context GQN E-GQN Target
Josh Tobin Geometry-Aware Neural Rendering
Examples
!18
Context GQN E-GQN Target
Josh Tobin Geometry-Aware Neural Rendering
Conclusion
!19
• Geometrically-inspired neural network primitives improve implicit 3D understanding
• Forcing the model to understand geometry can improve downstream robotic tasks
• How to go from this to general 3D understanding?