23270: AUGMENTED REALITY FOR NAVIGATION AND INFORMATIONAL ADAS

Sergii Bykov

Technical Lead Machine Learning

12 Oct 2017

Product Vision

Company Introduction

Apostera GmbH with headquarter in Munich, was established in March, 2017 together with 3 affiliated R&D centers to leverage 10+ years

engineering experience in complex software development for Automotive Industry.

Apostera GmbH engineering and business experience in Driver Experience, Navigation and Telecommunication domains together with unique

IP and mathematical talent guarantees creation of advanced product portfolio to bring mobility world to new era of autonomy.

Apostera GmbH today’s target is to reshape areas of Automotive Perception, Visualization, Path Planning, V2X and finally Autonomous Driving

in open and collaborative manner.

Perception: Advanced Surround View Monitoring, Software Smart Camera and Sensor Fusion

Visualization: Software Augmented Guidance (HUD and LCD)

Quality: A (AR, ADAS, AD) Testing Automated System Mobility: Software Managed Autonomous Driving

APOSTERA product lines - Basics

IA AAC

HAD

Informational ADAS

Active ADAS components

Highly AutomatedDriving

ADAS Platform

Representation For The Driver

Outdated

LCD screen

Smart Glasses

HUD in car

Real-depth HUD with wide FOV in car

Past

Alternative, fast developing market (today) On going development +2 years

Key Challenges For In-Vehicle AR

Usability – augmented reality subsystem should not disturb driver as it is continuously observed

Hardware limitations – computational, power consumption, zero latency (HUD)

Requirements for precise environmental model estimation for occlusion avoiding

Dependency on inaccurate map and navigation data

Distributed HW architectures, platform flexibility requirements

High precision absolute and relative positioning requirements

Components synchronization and latency avoidance

Embedded memory usage limitations, different memory models

Algorithms should be both configurable and efficient

Specific rendering requirements, not covered by general purpose frameworks

Variety of inputs under different platforms

Out-of-vehicle simulation (does not support natural simulation like classical navigation)

System Concept

Unique Automotive Augmented Reality Solution

Solution capable to create Augmented, mixed visual Reality for drivers and passengers based on Computer Vision, vehicle sensor, map data, V2X, navigation guidance using Data Fusion.

Automotive Cameras

Sensors/CAN

Navigation System/Map Data

Vehicle displaysProjection on wind shield - HUD

Telematics/V2XADAS

PlatformStep I

Integration of V2X information Motorbikes helmets

Path Planning and AR 360

ADASPlatform

In progress

ADASPlatform

Further Steps

Recognition and Tracking

• Road boundaries and lane detection

• Slopes estimation

• Vehicle recognition and tracking

• Distance & time to collision estimation

• Pedestrian detection and tracking

• Facade recognition and texture extraction

• Road signs recognition

Positioning

• Precise relative and absolute positioning

• Flexible data fusion and smooth Map Matching

• Automotive constrained SLAM

• Video-based digital gyroscope

Predictable Environmental Model, Safety Apps - V2X

• BSM transmitting/receiving

• Remote Vehicles trajectory prediction

• Basic safety applications based on collision detection

Integration with HD Maps

HD Maps utilization for Precise positioning, Map matching and Path planning, Junction assistance

Data generation for HD Maps

Contribution to ADAS attributes structure – NDS (HERE)

Augmented Reality

• LCD, HUD & further output devices

• Natural navigation hints & infographics

• Collison, Lane departure, Blind spots warnings, etc.

• POIs and supportive information (facades and parking slots highlighting, etc.)

Computer Vision Approaches

• Real-time feature extraction from video sensors

• Road scene semantic segmentation

• Adaptability and confidence estimation of output data

• GPU optimization for different platforms

Sensor Fusion

• Flexible fusion of data from internal and external sources

• LIDAR data merging

• 3D-environment model reconstruction based on different sensors

• Latency compensation & data extrapolation

Machine Learning Specifics

• CNN and DNN approaches

• Supervised MRF parameters adjustment

• CSP-based structure & parameters adjustment (both supervised and unsupervised)

• Weak classifiers boosting & others

Scientific and Engineering Expertise

System Overview

Live data from vehicle:- CAN data, Sensors- Video stream

ECU (e.g. Jetson TX2)

ADAS EngineSensor Abstraction LayerWeb InterfaceSW UpdateConfigurationDiagnostic

Video Stream withaugmented objects

ADAS/AR Engine

HUD/LCD

Head Unit

• Quick-install demonstration solution

• Platform for AR (allows to be portable)

• Integration with Head Units

• Integration with vehicle networks

• Using of own sensors if needed

Navigation data, preprocessedsensor data, etc.

Control/Settings

Perception Concept

Sensor Fusion: Data Inference

Optimal fusion filter parameters adjustment problem statement and solution developed to fit different car models

with different chassis geometries and steering wheel models/parameters.

Features:

Absolute and relative positioning

Dead reckoning

Fusion with available automotive grade sensors – GPS, steering wheel, steering wheel rate, wheels sensors

Fusion with navigation data

Rear movements support

Complex steering wheel models identification. Ability to integrate with provided models

GPS errors correction

Stability and robustness against complex conditions – tunnels, urban canyons

Sensor Fusion: Advanced Augmented Objects Positioning

Solving map accuracy problems

Placing:

•Road model

•Vehicles detection

•Map data

Position clarification:

• Camera motion model:•Video-based gyroscope

•Positioner Component

• Road model

• Objects tracking

Sensor Fusion: Comparing Solutions

Update frequency ~15 Hz (+extrapolation with any fps) Update frequency ~4-5 Hz

Apostera solution Reference solution

Lane Detection: Adaptability and Confidence

Lane Detection: 3D-scene Recognition Pipeline

Low level invariant features

• Single camera

• Stereo data

• Point clouds

Structural analysis

Probabilistic models

• Real-world features

• Physical objects

• 3D scene reconstruction

• Road situation

3D space scene fusion (different sensors input)

Backward knowledge propagation from high levels

Vehicle Detection

Convolutional neural network for vehicle detection

GPU Acceleration – CUDA

Running real-time on NVidia Jetson TX2

Inference speedup on embedded (TX2) GPU vs CPU is ~3x

More potential with new libraries (e.g. TensorRT)

Training speedup on desktop GPU vs CPU is ~20x

Classifier accuracy (about 50k, 960x540, ~55-60 deg

HFOV)

• Positive: 99.65%

• Negative: 99.82%

Size of detection down to 30 pix, detection range of

about 60 m

Figure – Vehicle detection examples

Road Scene Semantic Segmentation

Deep fully convolutional neural network for

semantic pixel-wise segmentation

Road scene understanding use cases: model

appearance, shape, spatial-relationship between

classes

Inference speedup GPU vs CPU is ~3x

Figure – Road scene segmentation examples

HMI Concept

Rendering Component Structure

Figure – Rendering component

Augmented Objects Primitives

Barrier Lane Line

Lane Arrow FishboneStreet Name

Augmented Objects Primitives And HMI

Head Up Display Concept. HUD vs LCD

Hardware limitation

• HUD devices are rarely

available on market

• FOV and object size

Timings

• Zero latency

• Driver eye position

Driver perception

• Virtual image distance

• Information balance

HUD Image Correction (Dewarping)

Figure – Corrected imageFigure – Uncorrected image

Need to correct slight distortion in the HUD image

A custom warp map was made by taking an image of a test pattern that was projected by the HUD and recorded

by a camera

Demo Application (LCD)

Summary: Key Technology Advantages

Proved understanding of pragmatic intersection and synergy between fundamental theoretical results and final

requirements

Formal mathematical approaches are complemented by deep learning

Solid GPU optimization

Automotive grade solutions integrated with all the data sources in vehicle – data fusion approaches

High robustness in various weather and road conditions, confidence is estimated for efficient fusion

Closed loops designed and implemented to enhance speed and robustness of each component

Integration with V2X and various navigation systems

System architecture supports distributed HW setup and integration with existing in-vehicle components if

required (environmental model, objects detection, navigation, positioner etc.)

Hierarchical Algorithmic Framework design highly optimizes computations on embedded platforms

Collaboration with scientific groups to integrate cutting edge approaches

Sergii Bykov

Technical Lead Machine Learning

sergii.bykov@apostera.com

BRINGING MOBILITY WORLD TO

NEW ERA OF AUTONOMY