road infrastructure analysis using deep learning · asset inventory: o mapping of road / road...
TRANSCRIPT
Max Sesselmann, Alexander Bock, Andreas Grossmann | ERPUG 2019| Vilnius
Road Infrastructure Analysisusing Deep Learning
Max Sesselmann, Alexander Bock, Andreas Grossmann | ERPUG 2019| Vilnius
CONTENT
o About LEHMANN+PARTNER
o Data acquisition: I.R.I.S
o Data analysis: deliverables
o Data analysis: automation using Deep Learning
ABOUT LEHMANN+PARTNER
Max Sesselmann, Alexander Bock, Andreas Grossmann ERPUG 2019| Vilnius
ABOUT LEHMANN+PARTNER
About LEHMANN+PARTNER
o based in Germany (three offices)
o since 2017: part of Ginger group (France, UK, Poland, Germany…)
o main business segments:
o road condition assessment (condition survey)
o road inventory value assessment (consultancy)
o development of technological solutions
LP
DATA ACQUISITION: INTEGRATED ROAD INFORMATION SYSTEM
Max Sesselmann, Alexander Bock, Andreas Grossmann | ERPUG 2019| Vilnius
Data Acquisition Technology I.R.I.S
Integrated Road Information System
o Inertial positioning system
(Applanix POS LV 420)
o Fraunhofer Pavement Profile Scanner
o Fraunhofer Clearance Profile Scanner
o Camera system
o Fully calibrated and time synchronized
3D Asset Mapping
Clearance Profile Scanner (CPS)
3D Condition Survey
Pavement Profile Scanner (PPS+)
DATA ANALYSIS: DELIVERABLES
Max Sesselmann, Alexander Bock, Andreas Grossmann | ERPUG 2019| Vilnius
DATA ANALYSIS : OVERVIEW
Asset inventory:
o Mapping of road / road corridor information as spatial
geometries
o Surface materials/types
o Utilization of urban spaces
o Routing / traffic modelling (e.g. road2simulation)
o Inventory of trees, manholes, signs, …
o 3D measurements
o Clearance height
o Compliance with guardrail height
o 3D data as basis for BIM
surface condition
long. & transv. evenness
Max Sesselmann, Alexander Bock, Andreas Grossmann | ERPUG 2019| Vilnius
DATA ANALYSIS : OVERVIEW
Condition surveys: route bands & condition grades
o capture surface condition using high resolution
cameras and LiDAR
o compute road condition indicators
o Surface characteristics
o Evenness (e.g. IRI, rutting, …)
o post-processing to German standard:
o Computation of condition grades (excellent
to poor) based on indicators for surface
condition, evenness, skid resistance, …
o Ready to be used in PMS (scenarios, planning, …)
Max Sesselmann, Alexander Bock, Andreas Grossmann | ERPUG 2019| Vilnius
DATA ANALYSIS : OVERVIEW
Data management: LP-portal
o LP-portal is a web-GIS platform
o stores all project-related spatial data
o Trajectories
o Mapping results (areas, materials, …)
o Imagery
o 3D point clouds
o cadastre data, road network, …
o Enables user to analyse data on his own:
o Measurement tools
o Visualization tools
DATA ANALYSIS: AUTOMATION USING DEEP LEARNING
Max Sesselmann, Alexander Bock, Andreas Grossmann | ERPUG 2019| Vilnius
AUTOMATED DATA ANALYSIS
Automation: What? Why? How?
o What: simple, repetitive mass data processeso Why: time-consuming, cost-intensive,
subjective/error-proneo How: Deep Learning
(state-of-the-art technology for complex computer vision problems)
„A machine-learning system is trained rather than explicitly programmed. It‘s presented with many examples relevant to a task, and it finds statisticalstructure in these examples that eventually allows the system to come up
with rules for automating the task.“ (CHOLLET 2018)
Max Sesselmann, Alexander Bock, Andreas Grossmann | ERPUG 2019| Vilnius
AUTOMATED DATA ANALYSIS
Automated image analysis: the ASINVOS approach
o ASINVOS project (2016-2018):
o starting point of Deep Learning technology in L+P’s workflows
o cooperation with Ilmenau University of Technology
o associated partner: Federal Highway Research Institute (BASt)
o Goals:
o … get faster results
o … get more objective results
o … get results in a higher quality
Recieved fundingNr.: 01IS15036
Max Sesselmann, Alexander Bock, Andreas Grossmann | ERPUG 2019| Vilnius
AUTOMATED DATA ANALYSIS
Automated image analysis: training of the ASINVOS network
o First step: intensive manual labelling (masks with semantic class information)
o Network learns characteristics from massive data examples (training)
Input image Labelled image
Input
Output
neural network structure(not trained)
crackno crack
backp
rop
agation
DURING TRAINING:
RESULT OF TRAINING: neural network that
“knows“ how to detecta crack
Max Sesselmann, Alexander Bock, Andreas Grossmann | ERPUG 2019| Vilnius
AUTOMATED DATA ANALYSIS
Automated image analysis: ASINVOS results
o primary result: confidence-map (heat-map) for a damage type (e.g. cracks)
o secondary results: damage contours (vectorized), analysis grids (German standard), …
Max Sesselmann, Alexander Bock, Andreas Grossmann | ERPUG 2019| Vilnius
AUTOMATED DATA ANALYSIS
Automated image analysis: ASINVOS results
o primary result: confidence-map (heat-map) for a damage type (e.g. cracks)
o secondary results: damage contours (vectorized), analysis grids (German standard), …
Max Sesselmann, Alexander Bock, Andreas Grossmann | ERPUG 2019| Vilnius
AUTOMATED DATA ANALYSIS
Automated image analysis: ASINVOS results
o primary result: confidence-map (heat-map) for a damage type (e.g. cracks)
o secondary results: damage contours (vectorized), analysis grids (German standard), …
Max Sesselmann, Alexander Bock, Andreas Grossmann | ERPUG 2019| Vilnius
AUTOMATED DATA ANALYSIS
Automated object detection from camera imagery
o Automated image segmentation: pixelswise classification of the image & contour-sharp delineation of
objects
o Variety of well-tested architectures available with very high scores on Cityscapes benchmark dataset
o Applications: detection of mobile/static objects, mapping, inventory/classification of road signs, …
skytraffic lightpole objectsign
roadsidewalkvehiclebuilding
vegetationgroundperson
license plate recognition using YOLO-V3
Prof. Dr.-Ing. Andreas Grossmann | ERPUG 2019| Vilnius
AUTOMATED DATA ANALYSIS
Example application 1: Anonymization of image data (data protection and privacy)
o Context: General Data Protection Regulation conformity
o images to be anonymized (example for 2019):
o > 20 million images from our own systems
o > 5 million images from external service providers
o 10 employees would work 347 days (~ 4 sec./img.)
o automated workflow takes ~ 0.4 sec./img.
(single instance of the neural network → scalable!)
o History of development:
o Beginning: based on bounding boxes
o Now: based on semantic segmentation
blurring of whole pedestrians and vehicles
Prof. Dr.-Ing. Andreas Grossmann | ERPUG 2019| Vilnius
AUTOMATED DATA ANALYSIS
Example application 2: automated mapping
o Deep Learning based image segmentation / object detection
o 3D data from CPS Scanner projected into camera: range image
o map objects automatically for further analysis in GIS
camera image image segmentation raw/optimized range image
mapped traffic sign in GIS
Max Sesselmann, Alexander Bock, Andreas Grossmann | ERPUG 2019| Vilnius
AUTOMATED DATA ANALYSIS
Example application 3:
automated distress mapping
o Deep Learning based damage detection
o Range image allows to precisely locate the
detected damages in GIS
o map damages automatically for further analysis
in GIS
camera image damage detection/classification
camera image damage detection/classification
lane markingcracksealed crackpatch
Max Sesselmann, Alexander Bock, Andreas Grossmann | ERPUG 2019| Vilnius
AUTOMATED DATA ANALYSIS
Example application 3: Mapping results from a sequence of images (GIS geometries)
Max Sesselmann, Alexander Bock, Andreas Grossmann | ERPUG 2019| Vilnius
CONCLUSION & REFERENCES
Conclusion
o I.R.I.S mobile mapping system allows to capture very detailed road information for large scale applications
o high precision active LiDAR instruments (Fraunhofer) and passive sensors (camera systems) integrated
o All data is georeferenced: very precise accuracy due to high-performance positioning system and system
calibration
o LEHMANN+PARTNER offers equipment, software and expertise for both asset inventory and condition
surveys
o Deep Learning helps in our workflows to get results faster and decide more objectively
o Outlook: ASFaLT-project → development of a Deep Learning based, fully-automated condition evaluation for
the specific needs of D-A-CH
Prof. Dr.-Ing. Andreas Grossmann | ERPUG 2019| Vilnius
CONCLUSION & REFERENCES
EISENBACH, M., STRICKER, R., SEICHTER, D., AMENDE, K., DEBES, K., SESSELMANN, M., EBERSBACH, D., STOECKERT, U. & H.-M. GROSS (2017): How to Get Pavement Distress Detection
Ready for Deep Learning? A Systematic Approach. Int. Joint Conf. on Neural Networks (IJCNN), Anchorage, USA, pp 2039 – 2047, doi: 10.1109/IJCNN.2017.7966101
EISENBACH, M., STRICKER, R., SESSELMANN, M., SEICHTER, D. & H.-M. GROSS (2019). Enhancing the quality of visual road condition assessment by Deep Learning.
World Road Congress 2019.
CHOLLET, F. (2018): Deep Learning with Python. Manning.
GROTHUM, O. (2019): Classification of Mobile-Mapping-Pointclouds Based on Machine Learning Algorithms. AGIT – Journal für Angewandte Geoinformatik 5-2019, pp. 315 – 328.
SEICHTER, D., EISENBACH, M., STRICKER, R. & H.-M. GROSS (2018): How to Improve Deep Learning based Pavement Distress Detection while Minimizing Human Effort. Proc. Int. Conf. on
Automation Science and Engineering (CASE), München, Germany, pp. 63-70, IEEE 2018
SESSELMANN, M., STRICKER, R. & M. EISENBACH (2019): Deep Learning for Automatic Detection and Classification of Road Damage from Mobile LiDAR Data. AGIT – Journal für
Angewandte Geoinformatik 5-2019, pp. 100 – 114.
STRICKER, R., EISENBACH, M., SESSELMANN, M., DEBES, K. & H.-M. GROSS (2019): Improving Visual Road Condition Assessment by Extensive Experiments on the Extended GAPs Dataset.
In: International Joint Conference on Neural Networks (IJCNN) 2019.
THANK YOU!