sparse volumetric representation of time-lapse point … · 2017-05-19 · 3 introduction captured...
TRANSCRIPT
![Page 1: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/1.jpg)
Innfarn Yoo, 05.08.2017
SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT CLOUD
![Page 2: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/2.jpg)
2
AGENDA
Introduction
Previous Work
Method
Result
Future Work
![Page 3: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/3.jpg)
3
INTRODUCTION
Captured external point cloud data by captured Kespry
- Drone Captured Photogrammetric Point Cloud
- Captured every 2-3 days
- 235 captures, 190 GB (avg. 810 MB)
- Each capture has 300 MB ~ 1.9 GB
- 10 ~ 50 million points
- Resolution is 10 ~ 20 cm
- Some noise
Time-lapse Point Cloud Dataset
![Page 4: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/4.jpg)
4
![Page 5: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/5.jpg)
5
INTRODUCTION
Capturing internal point cloud data
- Laser Scan (LIDAR) Point Cloud
- Captured every 2 weeks
- 23 captures, 510 GB (avg. 22 GB)
- Each capture has 13 ~ 45 GB
- 0.9 ~ 1.9 billion points
- Resolution is 1 mm ~
- Accurate (some noise near glass)
Time-lapse Point Cloud Dataset
![Page 6: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/6.jpg)
6
![Page 7: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/7.jpg)
7
INTRODUCTION
- Drone captured point cloud
- Dynamic loading and rendering of small, but many point cloud data
- 10 ~ 30 millions of points per capture
- We already showed our methods in GTC 2016
- Laser scan point cloud data
- Around 1.7 billions of points per capture
- 1.7 billions of points * 16 bytes (float x, y, z and color r, g, b, a) = 25.9 GB
- NVIDIA Quadro P6000 has 24 GB, GDDR5X memory, but not enough
Problems
Not a topic of this presentation
![Page 8: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/8.jpg)
8
INTRODUCTION
- Visualizing laser scan dataset in real-time
- Compactly save time-lapse laser scan dataset
- Provide more spatial information
- Primitive conversions
- Convert to machine learning friendly dataset
- Filling gaps in between points
Goals
Sparse Volumes
![Page 9: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/9.jpg)
9
AGENDA
Introduction
Previous Work
Method
Result
Future Work
![Page 10: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/10.jpg)
10
PREVIOUS WORK
- Point Cloud VR
- Time-Lapse VR Rendering
- Octree-based dynamic loading and rendering (LOD)
- Achieved 90 fps per eye
- Show entire dataset in VR village
GTC 2016
Render point cloud per two eyes, Markus Schuetz
![Page 11: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/11.jpg)
11
PREVIOUS WORK
- Progressive Blue-Noise Point Cloud
- Generating progressive blue-noise point cloud
- Buffer management using OpenGL 4.5 extension
- Dynamic loading and rendering of massive scale point cloud
GTC 2016
![Page 12: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/12.jpg)
12
PREVIOUS WORKDrone Captured Time-lapse Point Cloud Visualization
![Page 13: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/13.jpg)
13
AGENDA
Introduction
Previous Work
Method
Result
Future Work
![Page 14: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/14.jpg)
14
TIME-LAPSE LASER SCAN POINT CLOUD
- Time-lapse laser scan point cloud
- Notoriously big data size
- Capturing same space different time
- Some area has higher density than other
Pros & cons
![Page 15: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/15.jpg)
15
![Page 16: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/16.jpg)
16
SPARSE VOLUMETRIC REPRESENTATION
- Sparse Volumetric Representation
- Data compression
- Naturally represented by octree structure
- Voxel allows several algorithms
- e.g., Surface Extraction, Feature Detection, Object Detection
- Gives spatial relationship between voxels
- Can access neighbor voxels
Advantages of Sparse Volume
![Page 17: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/17.jpg)
17
CREATING SPARSE VOLUMEOffline Processes
Input laser scan files -
format: E57, LAS, or LAZCalculate Bounding Box
Generate Octree &
Splatting Points
Voxelate & Save Voxels Merge & Compress Voxels
![Page 18: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/18.jpg)
18
CREATING SPARSE VOLUME
- Make Octree power of 2 cube
- All leaves have same depth same volume for leaf node
- Subdivide leaf node into small subsets
- e.g., 1cm x 1cm x 1cm voxel
- Calculate whether points are in the voxel
- If a point hits the voxel, voxel activated (sparse)
Bounding Box Octree Voxels
![Page 19: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/19.jpg)
19
CREATING SPARSE VOLUME
- Activated voxels are only represented by several index bits (x, y, z)
- 202.42 meter x 226.53 meter x 74.67 meter area
- Voxelated by 1 cm x 1 cm x 1 cm
- Only 43 bits are required to save 1 voxel index x, y, & z
Octree Voxels Compression
![Page 20: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/20.jpg)
20
OCTREE-BASED SPARSE VOLUMES
- Save time-lapse point cloud
- If a voxel is activated, then only save colors
- System memory is not enough (out of core design)
- Process each laser scan
- Save to disk
- Merging and compression on disk
Merge & Compress Voxels
![Page 21: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/21.jpg)
21
SPARSE VOLUMETRIC REPRESENTATIONVoxelization
![Page 22: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/22.jpg)
22
RENDERINGProgressive Rendering
- More than 1 billions of points or voxels are too big to render in real-time
- Keep 60 fps, 80 millions of points per frames is maximum in NVIDIA Quadro P6000
- To see entire voxels or points, we use progressive rendering
- Usually used for physically-based rendering
![Page 23: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/23.jpg)
23
RENDERING
i. Not clean depth and color framebuffer per frame
i. Only clean when camera moved or rendering option changed
ii. Planning 80 millions of points budget per frame
i. Calculate view-frustum & octree node distance
ii. Calculate probability (visibility) per node based on distance
iii. Consecutively render additional points per frame per node
i. When no more points are remained in a node, then give remaining point budget to farther node
iv. Copy framebuffer to back buffer every frame
Progressive Rendering
![Page 24: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/24.jpg)
24
![Page 25: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/25.jpg)
25
![Page 26: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/26.jpg)
26
RENDERING
- Dynamic loading
- Planning how many points we can load per second (depending on disk speed)
- Calculate probability based on node distance from space & time
- Probability whether points need to be rendered in future frame
- Load points in different thread
- Sparse buffer allows to handle the points in virtual linear address
- Actual physical memory in GPU will be used when needed
- Loading or unloading blocks of sparse buffer based on spatiotemporal location
Thread & Sparse Buffer
![Page 27: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/27.jpg)
27
AGENDA
Introduction
Previous Work
Method
Result
Future Work
![Page 28: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/28.jpg)
28
RESULTS
0
0.5
1
1.5
2
2.5
5/8/2016 6/8/2016 7/8/2016 8/8/2016 9/8/2016 10/8/2016 11/8/2016 12/8/2016 1/8/2017 2/8/2017 3/8/2017
Num
ber
of
Poin
ts &
Voxels
(Billion)
Laser Scan Capture Date
Number of Points & Voxels (Voxelated result)
Num Points Num Voxels
![Page 29: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/29.jpg)
29
AGGREGATE RESULTS
0
50
100
150
200
250
300
350
400
Points Voxels (1cm) Remove Duplicated Voxels
0
5
10
15
20
25
30
Point to Voxel Conversion Size Comparison
Number of Object (billion) File Size (GB)
Unit: Billion Unit: GB
![Page 30: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/30.jpg)
30
RESULT
- Sparse voxel representation can alleviate notoriously big data size problem
- Preprocessing takes long time
- Several hours to process 400 GB laser scan data
- Progressive rendering allows see entire data set with real-time control
Overall
![Page 31: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/31.jpg)
31
AGENDA
Introduction
Previous Work
Method
Result & Demo
Future Work
![Page 32: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/32.jpg)
32
FUTURE WORK
- NVIDIA’s GVDB
- Similar to OpenVDB, but CUDA-based VDB
- Our dataset is larger than the limit of GVDB for now
- We have 3.5 billion voxels
- Later we will cut subset of voxels and process in GVDB
GVDB
Partially Converted to GVDB and Rendered using NVIDIA OptiXSplatting 10 GB of points to GVDB
![Page 33: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/33.jpg)
33
FUTURE WORK
- Integration to NVIDIA’s new ProViz viewer and editor
- ProViz team is developing new ProViz viewer and editor
- This work will be integrated
- Object Detection from Volumetric Point Cloud
- Detect objects using machine learning in 3D space
ProViz tool & Machine Learning
![Page 34: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/34.jpg)
34
NVIDIA’S NEW BUILDING
![Page 35: SPARSE VOLUMETRIC REPRESENTATION OF TIME-LAPSE POINT … · 2017-05-19 · 3 INTRODUCTION Captured external point cloud data by captured Kespry-Drone Captured Photogrammetric Point](https://reader036.vdocument.in/reader036/viewer/2022081407/5f87f06f449e732f87365607/html5/thumbnails/35.jpg)