sm3d: real-time simultaneous mapping and 3d...

SM3D: Real-Time Simultaneous Mapping and 3D Detection

Group Member: Guanqing Li, Rui Yang, Runfa Li, Shuangquan Sun

Mentors: Prof. Trivedi, Akshay Rangesh

Presentation Date: 06/02/2020

• Structure from Motion Module - Unsupervised• 3D Detection Module – Supervised

Input: RGB Images: 𝐼!, 𝐼!"#, 𝐼!"$Depth Images: 𝐷!"#GPS Location: 𝑃!"#, 𝑃!"$

Output:Mapping: 𝑀𝑎𝑝% 𝑡𝑜 𝑀𝑎𝑝!Depth Images: 𝐷!3D Bounding Box: 𝐵𝑜𝑥! 2

METHODOLOGY

• Expected goal

Ø Real-Time simultaneous mapping & 3D detection

• Novelty

Ø First time combining Structure from Motion (SFM) with 3Ddetection using Pseudo Lidar point cloud

• Significance

Ø A real-time system with diverse information (Mapping, 3Dbounding box, depth image, Pseudo Lidar point cloud) whichleaves slots for many further modification and improvement

3

EXPECTED GOALS, NOVELTY & SIGNIFICANCE

SIFT SuperPoint network Libviso2 built-in function

Original SFM learner [1] Advanced SFM learner

• KITTI sequence09: 0-700 frame

• Blue line: estimated trajectory

• Red line: GT trajectroy

4

CURRENT PROGRESS: SFM

CURRENT PROGRESS: SFM

• Complete KITTI sequence 09: 1590 frames

• Blue line: estimated trajectory

• Red line: GT trajectroy

Original SFM Learner [1] Advanced SFM Learner(this work)

5

• ATE: Absolute trajectory error

• RE: Rotation Error

• Advanced VS Original

REadv ≈ REori

ATEadv < ATEori

ATE RE

Original 0.0100 0.0016

Advanced 0.0091 0.0017

KITTI 09

KITTI 10

ATE RE

Original 0.0085 0.0016

Advanced 0.0081 0.0018

6

CURRENT PROGRESS: SFM

7

CURRENT PROGRESS: PSEUDO-LIDAR POINT CLOUD

Image Ground Truth Depth Map

Pseudo-LiDAR Point Cloud

Predicted Depth Map (Need improvement)

8

CURRENT PROGRESS: 3D DETECTION

9

It-1I2I It

Vt-1 VtV2V

V2I I2V

Image:

Velodyne:

• V2I : Velodyne → Reference → Rectified → Camera → Image(computationally expensive)

• V2V = I2I(If no relative motion between Lidar & Camera)

3D 3D 3D 3D 2D

CURRENT PROGRESS: POINT STACKING ALGORITHM

10

Stack Number = 3

• Keep the total point number same after “Stack Number” by removing the earliest points

time: t = 0 t = 4t = 3t = 2t = 1 t = T

V0 V1 VTV2 V3 V4Velodyne: …Net point number: n0 n1 n2 n3 n4 nT

Total point number: n0 N1 = n0+n1 N = N1 +n2 N =(N-n3)+n3 N =(N-n3)+n3 N =(N-nT)+nT

CURRENT PROGRESS: POINT STACKING ALGORITHM

CURRENT PROGRESS: POINT STACKING RESULT

11

Pseudo-LiDAR Point Cloud before Augmentation Pseudo-LiDAR Point Cloud after Augmentation(Stack number = 2)

CURRENT PROGRESS: REMAINING ISSUES

12

• Improve the quality of predicted depth mapsØ Choose appropriate hyperparameters to avoid overfitting of disparity lossØ Now are using GT depth for following stepØ Quality of the depth image decide the performance of Lidar & 3D detection

• Train Pointpillar, include 3D detection into the architecture while maintain efficiency.Ø Mapping alone 68 ms/frame on week 6,

182 ms/frame this week (same code, same environment, server busy)

Disparity Loss Pose Loss Total Loss

INDIVIDUAL WORKLOAD

13

Ø Depth map and projection matrix: Runfa

Ø Depth to pseudo Lidar point clouds: All

Ø 3D detection on pseudo lidar point clouds

PointPillars (high speed): All

Debugging and documentation: Shuangquan

Ø Stacking Pseudo-LiDAR Point Clouds: Runfa

MILESTONES & TIMELINE

14

• Week 10:

Ø 3D detection

• Week 11:

Ø Report Writing

Ø Add supervision to 3D detection, end-to-end multi-task learning (flexible)

REFERENCE

[1] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe. Unsupervised learning of depth and

ego-motion from video. In CVPR, 2017. 1, 3, 6, 7, 8

[2] Y. Wang, W.-L. Chao, D. Garg, B. Hariharan, M. Campbell, and K. Q. Weinberger.

Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for

autonomous driving. In CVPR, 2019.

[3] Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar

Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR,

2019. 1, 2, 6, 7, 8, 14, 15, 16

15

THANKS

QUESTIONS?