slam@nvidia - silvio savarese · we have done kinectfusion-type of processing using softkinetic...
TRANSCRIPT
![Page 1: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/1.jpg)
SLAM@NVIDIA Kari Pulli| Senior Director of Research
![Page 2: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/2.jpg)
Overview
! Keyframe-based SlAM
! 3D rendering for Augmented Reality
! Problems with traditional keyframe-based SLAM
! Solution: Deferred Triangulation SLAM
![Page 3: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/3.jpg)
KeyFrame-based SLAM
3D Mapping [Bundle Adjustment]
2D Tracking [Optical Flow]
3D Tracking [Pose Estimation]
Stereo Initialization [Triangulation]
Rendering [Overlaying (AR)]
[Scene Reconstruction]
Optical flow
Stereo triangulation
3D Tracking and pose estimation
BA
Incremental mapping and camera pose refinement
Time
Tracking
Mapping
Rendering Rendering objects with the camera poses and geometry (map)
Bundle Adjustment
Adding Keyframes, data association, and recovery
BA BA BA BA
![Page 4: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/4.jpg)
Tracking example
![Page 5: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/5.jpg)
![Page 6: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/6.jpg)
![Page 7: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/7.jpg)
![Page 8: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/8.jpg)
![Page 9: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/9.jpg)
![Page 10: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/10.jpg)
![Page 11: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/11.jpg)
![Page 12: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/12.jpg)
DTAM
![Page 13: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/13.jpg)
![Page 14: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/14.jpg)
! We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better
than on Tango
![Page 15: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/15.jpg)
![Page 16: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/16.jpg)
![Page 17: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/17.jpg)
![Page 18: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/18.jpg)
![Page 19: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/19.jpg)
How to deal with the rotation?
ISMAR 2013
ISMAR 2012
![Page 20: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/20.jpg)
This is how
3DV 2014
![Page 21: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/21.jpg)
How to deal with the rotation?
! Deferred triangulation
0.5x Speed for visualization
Deferred 2D points
Triangulated 3D points
![Page 22: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/22.jpg)
How to deal with the rotation?
! Deferred triangulation
! Jointly (2D/3D) constrain a pose
0.5x Speed for visualization
Deferred 2D points
Triangulated 3D points
![Page 23: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/23.jpg)
How to overcome the rotation?
! Deferred triangulation
! Jointly (2D/3D) constrain a pose
! Region merging
![Page 24: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/24.jpg)
Pose estimation
![Page 25: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/25.jpg)
Epipolar segment
![Page 26: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/26.jpg)
Epipolar segment
![Page 27: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/27.jpg)
Pose estimation
![Page 28: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/28.jpg)
Bundle Adjustment
![Page 29: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/29.jpg)
![Page 30: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/30.jpg)
![Page 31: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/31.jpg)
Quantitative evaluation
![Page 32: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/32.jpg)
![Page 33: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/33.jpg)
![Page 34: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/34.jpg)
![Page 35: SLAM@NVIDIA - Silvio Savarese · We have done Kinectfusion-type of processing using SoftKinetic range scanners, the quality and framerate of the depth is better than on Tango . How](https://reader034.vdocument.in/reader034/viewer/2022050217/5f638684c4137966175d410d/html5/thumbnails/35.jpg)
Summary
! Keyframe-based SLAM is efficient
! and can run in real time on mobile devices
! But it has problems
! A separate initialization phase is annoying
! Breaking with pure rotations is a critical failure
! Both can be addressed by
! tracking first in 2D
! deferring triangulation until there is enough baseline between the keyframes
! Bonus: we plan to open source the implementation