the engineering challenges of autonomous driving · simulation & safety validation trends:...

The Engineering Challenges of

Autonomous Driving

October 11, 2018

2

OUR PANELISTS

Speakers:• Phil Magney

Founder & PrincipalVSI Labs

• Matthew LinderAV Solutions EngineerVSI Labs

• Chris PoschDirector of Engineering – AutomotiveFLIR Systems

Moderator:• Kris Keach

Design News

2

Engineering Challenges of Automated Driving

Technical research for those designing or developing automated vehicle technologies

• Applied research on AV technologies since 2014

• Offers technical advisory and research services to support R&D & planning for automotive, suppliers and technology industry

• VSI’s applied research includes functional examinations on HW/SW components or systems.

• VSI has it own fleet for research purposes

• Skilled in contemporary automotive and AV technologies including connectivity, simulation, programming, computer vision, neural networks (AI), and ISO 26262

4

Who is VSI

Copyright 2018 – VSI Labs

Opening Remarks

• Automation = Safety

• Born from active safety systems

• Partial automation (1-3) as in series production becomes a big selling feature over the next 10 + years before fleet automation starts to dent private ownership

• High automation (L4/5) starting to emerge in next few years

• Gradual deployment in major metros but within a highly constrained operating domain!

• Infrastructure will generally lag for many years… some elements will go to the cloud!

5Copyright 2018 – VSI Labs

The Tech Behind Automated Vehicle Systems

SensorsProcessorsSoftware

6

Vehicle Setup

NW/CAN Interfacing

AV Domain Controller

Lidar Stack

Radars Camera &

Vision

Control Systems

Calibrations

Cabin Sensing

Localization Assets

Connectivity

Lane and Object

Detection

Object Positions & Movements

3D Object & Classification

Lane models. Object-based Localization

OTA/V2XDriver

Readiness

The Componentry of Automated Vehicles


IMU/GNSS

Odometry &Correction

By-Wire Control Systems

Lane KeepingSign rec.

Pedestrian

ACCObject

Detection

Environment Mapping

Positing and Trajectory

Precision Lane Level

Localization

Firmware Updates

SPaT conn.

Driver Engagement & Attention

Timing and Synchronize

Level of Automation L2+

8

Automation Pipeline

Latitudinal Control

Longitudinal Control

Image Sensors• Camera(s)• Thermal • Scanning LiDAR

Ranging Sensors• Radar• Flash LiDAR• Ultrasonic

Localize & Plan • Sensor fusion • Localization• Path planning

Control Algorithms• Open interface• Control

Algorithms• PID Control

Vehicle Tasking

Event Planning

Odometry • IMU• RTK• Precision map

Behavior• Trajectories/Velocities • Detection/Avoidance• Maneuver planning• Fail safe plan

Path Planning

Control Fail Safe

Environmental Modeling

Sensor Fusion


Automated Vehicle Systems – Passive Camera Vision

• Best sensor for classifying objects and scenes• Poor at range and speed• Computationally expensive


Automated Vehicle Systems – Thermal Camera Vision

• Good sensor for identifying humans in challenging light conditions

• Performs similar in day or night and can see in total darkness

• Good for cluttered environments, especially pedestrians.

• Best when coupled with other sensors


Passive Active

Source: FLIR

Automated Vehicle Systems – LiDAR Imaging

• LiDAR is best for 3D perception & environmental modeling• Limited performance in poor weather• Very expensive still… some promising developments though!


Automated Vehicle Systems – Radar

• Radar is best for “ranging” – judging distances and movements of its “targets”

• 4 dimensions (elevation, range, doppler and angle)

• Complimentary for sensor suites



Developing Automated Vehicle Systems – The “AV Stack”

14

Challenges & Limitations

Improving Performance & Safety

15

• Sensors• Camera, Radar, Lidar, etc.

• Localization Assets • Localization Assets (Maps and Metadata)• RTK Support

• Inertia Measurement (Odometry)• Adjusted Control Algorithms

• V2X• V2V (car to car), V2I (signals and roadside),

V2P (to pedestrians)

• Redundancies

• Fail Safe Strategies

• Tele-Operation

How AVs will See in Poor Weather?


Virtual Infrastructure

• Physical Infrastructure will transition to virtual infrastructure

• Painted lanes lines give way to MAP-based lane models

• Signaled intersections give way to digital Signal Phase and Timing (SPaT)

• Road signs give way to virtual signs (as this is stored as metadata in machine readable form)

• Lanes become reconfigurable without physical gates or dedicated lanes

• Construction zones transmit electronically (smart signs)


18

Limitations of Vision-based Automation


Actual trajectory

Proper trajectory

• A recent accident in Mountain View shows limitations of a vision based system.

• Poor lane markings and vast differences in surface reflectivity caused this accident.

• A precision map (with an HD lane model), could have eliminated this accident.

Limitations of Vision Based Solutions (Tesla Accident)


Localization

Improving Performance & Safety with Software

20

• Road model (ADAS Map) • Topology

• Routing

• Speed attributes, etc.)

• Lane Models • Lane geometry

• Polylines

• Trajectories

• Localization Layer • Landmarks

• Signs, barriers, poles etc.

• Edges and boundaries

• Voxels

• Confidence Index

21

MAP-based Localization Assets


• Point Cloud Localization -- compare a 3D point cloud from a Lidar sensor scan to an existing 3D point cloud. Begin the alignment process.

• Processor Heavy – You have to match every point using Iterative Closest-Point (ICP) algorithm which is very taxing from a compute standpoint.

22

Localization With High Definition Maps


• When executing point-cloud to point-cloud localization, the points in the point-cloud are unclassified points.

• In landmark-based approaches the objects are classified.

• The downside is that it's much more difficult and expensive to create these maps.

23

Source: HERE



• GNSS / Real Time Kinematics (RTK)

• Supplements the GPS known positions with correction stations.

• Does not work well in dense urban areas as you need line of site because of GPS fallout.

• You still need a lane model.

24

Source: VBOX

Localization With Correction Services


Sensor Stack

IMU

GPS

LiDAR

Localization Algorithms

Transform

Source: HERE

Positioning

Odometry

Camera

Radar

AV Processing Stack

By Wire Control Messages

Lane Detection Object Avoidance


Localization using precision maps requires real time processing of objects, images, scenes or other patterns against an embedded map


Developing Automated Vehicles

Testing & Simulation

26

Developing Automated Vehicles -- Defining Operating Domains

• USDOT-NHTSA federal guidance for Automated Driving Systems (ADS) require developers to defined their solution according to Operational Design Domains (ODD).

• A SAE Level 2, 3 or 4 vehicle could have one or multiple ODDs – e.g. geo-fenced urban, divided highways, automated parking, traffic jam assist, etc.

• AVs should be developed, tested and validated against all scenarios that could happen within the ODDs.

• Widely understood that AVs require millions if not billions to be proven safe enough!


Developing Automated Vehicles – Filling in the Gaps

28

All States and Scenarios

Field Tests/Real World Data Collection

Simulation

Test Tracks

Algorithm Improvement

It is recommended to utilize detailed simulations to develop the systems, test-track tests to validate components and full-vehicles, and field tests to verify the real-life system robustness.


Simulation & Safety Validation Trends: End-to-end Toolchain

Single supplier’s end-to-end simulation toolchain integration with AV stack

• An integrated solution with simulation tools necessary to test multiple scenes and edge cases

• Can speed up development and testing dramatically

• Includes predefined scenarios used by safety agencies such as Euro NCAP


Simulation & Safety Validation Trends: Standardized IF

• A standardized interface specification for simulations

• Soon virtual evaluation of automated driving functions will be required

• Test/simulation software tools must be certified

• New players on the automotive simulation market still have gaps: (i.e. vehicle dynamics and powertrain models)


Closing Remarks

• Automated vehicles requires a precision localization for performance and safety purposes. • Precision maps achieve this objective as the metadata in the maps enable the vehicle to better understand its position against

ground truth.

• Automated vehicles could benefit from correction services of one form or another but this is not seen as practical as the hardware and software is expensive!

• Also, does not work well in dense urban environments.

• The industry has tried brute force to solve the compute challenge but realizing this is not efficient either • Companies are developing ASICs to cope with the huge compute requirements pushing some of the functional load to an

optimized ECU.

• Environmental sensors by themselves with cannot handle all situations• For example, long range radar is vital but you have to tune out non-doppler activity (static objects for example)

• Cameras get confused easily, especially with AI-based solutions • You need anomaly checkers, or deterministic rules to filter out the anomalies

• Challenging light conditions or cluttered environments are problematic

• Infrastructure is lagging • And may always be behind where it should be!

• Some infrastructure is going to cloud (mapping assets, localization assets, signage, etc.)


Contact

• Phil Magney

• VSI Labs

• [email protected]

• +1-952-215-1797

• VSI Web Site

Thermal Cameras in ADAS and AV

A U TO N O M O U S V E H I C L E T E C H N O L O G Y

W E B I N A R

Proprietary - Company Confidential ©2018 FLIR Systems Inc. Information and equipment described herein

may require US Government authorization for export purposes. Diversion contrary to US law is prohibited.

Thermal Cameras Make ADAS & AV Safer

Thermal provides a redundant imaging solution to fill the visible-camera performance gaps.

Ther

mal

Cam

eras

will

sig

nif

ican

tly

hel

p in

all

thes

e ca

ses



Reliably Classify in Cluttered Environments

Free starter thermal image dataset - over 14,000 annotated thermal images



www.flir.com/adas

Best sensor for pedestrian detection & reliable

classification

• Performs similar in day or night

• See in total darkness - 4x headlights

• Not blinded by the sun, shade, tunnels, etc.

• Sees through most of fog

• Deep learning can be readily applied to

thermal camera data

Collect data with FLIR ADK™ and FREE thermal starter dataset.

Person 0.84

FLIR Systems Inc.

• FLIR Thermal infrared cameras reliably classify objects – especially humans in a cluttered environment

• FLIR thermal infrared cameras see in challenging lighting conditions – flog, sun glare, complete darkness

• FLIR has the only auto-qualified thermal sensor and it is installed in >500K vehicles

Developing Imaging Solutions That Enhance Perception and Awareness

See in Total DarknessEnhanced Long Range

ImagingSee Through Obscurants

Non-Contact Temperature

Reliably ClassifyPeople & Animals

The World Leader in Thermal Infrared Imaging

40

THANK YOU!

• Please be sure to complete the feedback survey

• Thank you for attending today’s Design News

Editorial Event:

The Engineering Challenges of Autonomous Driving

40

the engineering challenges of autonomous driving · simulation & safety validation trends:...

Documents