2nd international workshop on point cloud processing stuttgart...
TRANSCRIPT
![Page 1: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/1.jpg)
Universität Stuttgart
ifpifp
Deriving Semanticsfrom Textured Meshes
2nd International Workshop on Point Cloud Processing
Stuttgart, December 04-05, 2019
ifpDominik Laupheimer
![Page 2: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/2.jpg)
ifpifpUniversität Stuttgart
Textured Meshes
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
A mesh is a wired point cloud.
2019/12/05 2
![Page 3: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/3.jpg)
ifpifpUniversität Stuttgart
• Reduced Memory Consumption
• Good Compression Behavior (Noise Reduction)
• Waterproof Surface Representation
Explicit Topology
Texture
• Unambiguous Normal Calculation
2019/12/05 3
Motivation: Why Meshes?
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
![Page 4: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/4.jpg)
ifpifpUniversität Stuttgart
• “Fusing” Data Representation
Imagery
Point Clouds
• Use-cases
Viewshed and Flood Analysis
City Models
Visualization + VR
2019/12/05 4
Motivation: Why Meshes?
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
![Page 5: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/5.jpg)
ifpifpUniversität Stuttgart
“Meshes are comprehensive maps for literally the whole world!”
Motivation: Why Meshes?
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
5
Snapshot of Google’s 3D representation (mesh).
![Page 6: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/6.jpg)
ifpifpUniversität Stuttgart
Aim: Semantic Segmentation of Meshes
Machine
Learning
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
6
![Page 7: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/7.jpg)
ifpifpUniversität Stuttgart
ALS Point Cloud Benchmarks / Public Data Sets Mesh Benchmarks / Public Data Sets
ISPRS 3D Semantic Labeling (Vaihingen, V3D)
RoofN3D (TU Berlin)
AHN3 (the Netherlands)
GRSS Data Fusion Contest
(Track 4: 3D Point Cloud Classification)
?
Availability of Ground Truth Data
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
7
![Page 8: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/8.jpg)
ifpifpUniversität Stuttgart
ALS Point Cloud Benchmarks / Public Data Sets Mesh Benchmarks / Public Data Sets
ISPRS 3D Semantic Labeling (Vaihingen, V3D)
RoofN3D (TU Berlin)
AHN3 (the Netherlands)
GRSS Data Fusion Contest
(Track 4: 3D Point Cloud Classification)
Only Indoor Scenes
Availability of Ground Truth Data
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
8
![Page 9: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/9.jpg)
ifpifpUniversität Stuttgart
Data Acquisition
Ground Truth Generation
2019/12/05
LiDAR: 800 pts/m², footprint Ø < 3 cm
Photogrammetry: GSD @ 3.7 mm (nadir), 2.3 cm (oblique)
Cramer, M.; Haala, N.; Laupheimer, D.; Mandlburger, G. & Havel, P. , 2018:
Ultra-high precision UAV-based LiDAR and Dense Image Matching. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XLII-1, 115-120.
DOI: 10.5194/isprs-archives-XLII-1-115-2018
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
9
![Page 10: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/10.jpg)
ifpifpUniversität Stuttgart
Ground Truth Generation
2019/12/05
✋✎
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
10
LiDAR Point Cloud
Textured Mesh
Labeled Mesh
![Page 11: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/11.jpg)
ifpifpUniversität Stuttgart
Labeled Ground Truth
2019/12/05
0. building mass/facade 1. roof 2. impervious surface 3. green space
4. mid and high vegetation 5. vehicle 6. chimney/antenna 7. clutter
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
11
0 1 2 3 4 5 6 7
![Page 12: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/12.jpg)
ifpifpUniversität Stuttgart
Aim: Semantic Segmentation of Meshes
Machine
Learning
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
12
![Page 13: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/13.jpg)
ifpifpUniversität Stuttgart
Methodology
2019/12/05
• Ground Truth Generation
• Feature Calculation
Geometric & radiometric
Multi-scale contextual features
• Train Classifier
Multi-Branch 1D CNN*
RF
• Inference/Evaluation
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
13
* George, D., Xie, X. & Tam, G., 2018:
3D Mesh Segmentation via Multi-branch 1D Convolutional Neural Networks. Graphical Models, 96, 1-10.
![Page 14: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/14.jpg)
ifpifpUniversität Stuttgart
Feature Calculation
Height above Ground
Density Horizontality
2019/12/05 14
Texture
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
![Page 15: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/15.jpg)
ifpifpUniversität Stuttgart
Network Architecture
Feature-Based Multi-Branch 1D CNN
2019/12/05
George, D., Xie, X. & Tam, G., 2018:
3D Mesh Segmentation via Multi-branch 1D Convolutional Neural Networks. Graphical Models, 96, 1-10.
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
15
Scale 0
Scale 1
Scale 2
![Page 16: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/16.jpg)
ifpifpUniversität Stuttgart
RF Prediction250 trees, depth: 25
Training Time: 6.6h
Accuracy: 79.01%
Inference Time: 45.83s
Results
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
16
Textured Mesh~300.000 faces
Ground TruthLabel Noise
1D CNN Prediction9.3 million parameters
Training Time: < 15min
Accuracy: 79.87%
Inference Time: 14.69s
![Page 17: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/17.jpg)
ifpifpUniversität Stuttgart
Feature-Based Multi-Branch 1D CNN
Comparison of Feature Influence
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
17
Textured Mesh Prediction
(all features)
Prediction
(geometry only)
Prediction
(texture only)
Ground Truth
![Page 18: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/18.jpg)
ifpifpUniversität Stuttgart
Feature-Based Multi-Branch 1D CNN
Comparison of Feature Influence
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
18
Textured Mesh Prediction Using
Geometric Features Only
Prediction Using
Geometric & Radiometric
Features
![Page 19: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/19.jpg)
ifpifpUniversität Stuttgart 2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
19
Conclusion
• Alternative 3D Data Representation: Meshes
Data fusion
Good georeferencing/relative orientation of LiDAR and imagery necessary
• Ground Truth Generation
• Pipeline for Mesh Generation and Semantic Segmentation
Overall accuracy: ~80 %
Detection of buildings and mid/high vegetation works well
• Features
Geometry > Radiometry
Radiometry matters!
![Page 20: 2nd International Workshop on Point Cloud Processing Stuttgart ...pcp2019.ifp.uni-stuttgart.de/presentations/12-PCP_2019_Laupheimer... · ifp DerivingSemantics fromTexturedMeshes](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03042d7e708231d4071f7e/html5/thumbnails/20.jpg)
ifpifpUniversität Stuttgart
• Ground Truth Generation
Avoid label noise
Crowdsourcing
• Georeferencing
Hybrid georeferencingGlira, P.; Pfeifer, N.; Mandlburger, G., 2019:
Hybrid Orientation of Airborne LIDAR Point Clouds and Aerial Images.
ISPRS Annals of Photogrammetry, Remote Sensing and
Spatial Information Sciences, Volume IV-2/W5, 2019, pp.567-574.
DOI: 10.5194/isprs-annals-IV-2-W5-567-2019
• Features
Incorporate LiDAR features
Incorporate texture explicitly
Future Work
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
20