1

Jace AllenBusiness Development Manager – Simulation, Test, and EEDM

dSPACE Inc. · 50131 Pontiac Trail · Wixom, MI · 48393 USA

Autonomous Automation: How do we get to a Million Miles of testing?

Agenda

1. Intro to V&V for ADAS/AV/HAD

Changing Environment of AV/HAD

ADAS V&V Process and ISO26262

2. Testing Toolchain for ADAS/AV/HAD

Models, Scenarios, and Sensors

HIL Testing and Sensor Fusion Needs

SIL Testing and Cluster Simulation

3. Testing Process and Autonomous Automation

Testing Methods and Tools

Real-time Testing and Observers

Test Management and Automation

Optimizing Testing

Challenges – Testing autonomous driving in real traffic

4

Unlimited number of real-life traffic scenarios

Many unknown factors, human driver no longer as a fallback … how to validate system robustness?

Exponential growth in testing effort …. hundreds of millions of test kilometers required.

Source: nik/pkbThe real world is complex The real world is unpredictable The real world is hazardous

Validate System Behavior with Simulation

5

MBD Testing = Simulation at All Levels

Advantages of simulation: reproducibility,

test beyond performance/endurance

limits and dangerous situations

ISO 26262 recommends MIL/SIL/HIL

simulation for conducting the software

safety requirements verification

What changes with testing Autonomous

Vehicles?

dSPACE-Internal6

ADAS/AV development process

System

conceptMIL

(traffic simulation)

Component HIL

(closed-loop)

ADAS HIL(closed-loop)

Test drives on

prov. ground

Test drives in

real traffic

Homolo-

gation

Prototyping

AlgorithmADAS/Sensor ECU(s)

under Test

Target

implementation

Requirements

Specification

SIL(closed-loop)

Driving

simulator

SIL(closed-loop)

Machine

learning

dSPACE-Internal7

ADAS/AV development process

System

conceptMIL

(traffic simulation)

Component HIL

(closed-loop)

ADAS HIL(closed-loop)

Test drives on

prov. ground

Test drives in

real traffic

Homolo-

gation

Prototyping

AlgorithmADAS/Sensor ECU(s)

under Test

Component HIL

(open-loop)

Target

implementation

SIL (open-loop,

data playback)

Requirements

Specification

SIL(closed-loop)

Driving

simulator

SIL(open-loop)

SIL(closed-loop)

dSPACE-Internal8

ADAS/AV development process

System

conceptMIL

(traffic simulation)

Component HIL

(closed-loop)

ADAS HIL(closed-loop)

Test drives on

prov. ground

Test drives in

real traffic

Homolo-

gation

AlgorithmADAS/Sensor ECU(s)

under Test

Component HIL

(open-loop)

Requirements

Specification

SIL(closed-loop)

SIL(open-loop)

Cloud

Driving

simulator

Machine

learningPrototyping

Target

implementation

SIL (open-loop,

data playback)SIL

(closed-loop)

PC cluster

PC cluster

Cloud

Measurement data(Sensor and vehicle data)

Test scenarios/cases

Model-, parameter-,

scenario management

9

Challenges – Changing validation process

Data & test management

Models, parameters,

tests, test results, …

• Vehicle, driver, sensors

• Road, road networks

• Traffic, roadside structures

• Environmental conditions (weather, …)

• Driving maneuvers

• …

Automated test execution,

test evaluation and control

Defined scenarios

Stochastic parameter variation

Test control to detect critical

scenarios based on metrics and

evaluation criteria

Requirements-based

testing not sufficient

Virtual ECUs Real ECUs

PC cluster

Traffic scenarios

(critical, representative, …)

Scenario databases, …

Support of open file

formats and standards

TESTING TOOLCHAIN FOR ADAS/AV/HAD

dSPACE Solutions: A Powerful ADAS Toolchain

11

Test

Scenarios

Visualize?

Maneuver

Control

&

Experiment

Test

Management

Tests

Simulate

Models

12

Automotive Simulation Models (ASM) for ADAS and autonomous driving

Road networks Roads and intersections

Lane support

Artificial/real world roads

Road import

Roadside structures

MotionDeskAnimation

ModelDeskParameterization

Traffic Objects Static and dynamic objects

Vehicles, trucks,

pedestrians

Traffic signs, traffic lights,

parking vehicles, …

…Environment Sensors 2-D/3-D sensors

Camera, radar, lidar, …

Line, lane and traffic sign

recognition

Object list simulation

Vehicle simulation Vehicle Dynamics

Drivetrain

Soft-ECU network

Driver model

Maneuver

Environment

Vehicle

Dynamics

Traffic

Test Scenario Definition

• ASM Traffic Scenario definition

• ASM Simulated Traffic flow and Tool automation

• Independent definition of fellow scenarios

• Demo scenarios for standards (Euro NCAP)

• Open API for Automation

13

Road Import

OpenDRIVE, OpenCRG

Measurement data/GNSS

Here/ADAS RP

OSM, Google Earth

Scenario Import

Manually defined based

on expert knowledge

GIDAS database

OpenSCENARIO (planned)

ModelDesk API – Stochastic, etc.

Real sensors

and “ADAS” ECUs

14

dSPACE HIL simulation technology for ADAS and autonomous driving

Road, environment,

driving maneuvers

Virtual driver

Vehicle dynamics

Sensor

models

Automotive Simulation

Models (ASM)

Real-time

HIL simulator

Soft

ECUs

Brake/engine torque,

steering angle, …

Real sensors

and “ADAS” ECUs

15

dSPACE HIL simulation technology for ADAS and autonomous driving

Road, environment,

driving maneuvers

Virtual driver

Vehicle dynamics

Sensor

models

Real-time

HIL simulator

Soft

ECUs

Sensor system

Automotive Simulation

Models (ASM)

16

ASM Traffic and sensor models – HIL or VEOS

Integration of sensor-specific error

models in Simulink possible due to

open Automotive Simulation Models

(ASM)

3-D object sensor

2-D contour sensor

Attribute sensor

Traffic

environment

Idealized

sensor data

Error model

(Noise, false

positive/negative

detections, …)

Sensor data

with realistic

errors

Simulink®

ASMSensor or customer

specific implementation

Netw

ork

Man

ag

em

en

t

Raw data

Target list

Object list

Sensor ECUSensor

Frontend

Application logic(Trajectory planning,

Motion control)

Object tracking

Detection, Data Proc.

Preprocessing

Options for testing Sensor ECUs

Option 1 & 2

Technology independent approach

Provide ideal ground truth based

information

Part of ASM

Calculated on SCALEXIO CN (CPU)

Option 3 & 4

Physics-based approach

More related to the measurement

principle of a sensor

Calculated on GPU

Option 5

Test Bench with real Sensor ECU

Confidential, Information are subject to change without notice

Opt. 2: Insert Object list

Opt. 1: Restbus

Vehicle network1) Automotive Real-Time Radar Scene Generator

Opt. 3: Insert target list

Opt. 4: Insert raw data

OTA-DeviceOpt. 5: Simulation over-the-air (OTA)

Netw

ork

Man

ag

em

en

t

Raw data

Target list

Object list

Sensor ECUSensor

Frontend

Application logic(Trajectory planning,

Motion control)

Object tracking

Detection, Data Proc.

Preprocessing

Confidential, Information are subject to change without notice

Options for testing Sensor ECUs

OTA-DeviceOpt. 5: Simulation over-the-air (OTA)

Opt. 1: Restbus

Opt. 2: Insert Object list

Opt. 3: Insert target list

Opt. 4: Insert raw data

Vehicle network

Option Camera Radar Lidar Ultrasonic

1

2

3 Under

development n/a n/a

4 Under

development3D point cloud n/a

5 ARSG1) n/a

1) Automotive Radar Scene Generator

Raw Sensor Data Generation –

Environment Sensor Interface Unit

Powerful high-end FPGA Xilinx Zynq UltraScale+ (MPSoC ZU9)

Synchronous output for >8 sensors (Cameras/Radar1/Lidar1) with 1 ESI Unit2

Up to 15.9 Gbit/s aggregated data rate

FPGA model (partly) open for customers (VHDL, Verilog, XSG, and HLS)

Flexible adaption of video interfaces via plug-in modules (FMC)

Direct camera I/F (GMSL, FPD-Link III, Ethernet AVB1 & GigE Vision1)

Customer-specific interfaces

ESI Unit Plug-on Device (POD) for short range video interfaces

parallel, HiSPI, CSI2, LVDS, etc.

High dynamic range (HDR) support: Up to 20 bit1

Feedback channel to ECUs (e.g. field of view, exposure time, …) from ESI Unit

Both Open loop and closed loop testing with the ESI1Under Development

2Depending on customer setup

FMC modules

Confidential

Camera Image Sensor

Source: http://www.onsemi.com/

Source: https://en.wikipedia.org/

Register InterfaceI²C

Bayer Pattern (RGGB)

Lens ImagerImage

processing

ESI Unit: FPGA Firmware Overview

Confidential

Video

Input

Environment Sensor Interface Unit (Xilinx Zynq ZU9EG FPGA)

HDMI Input

Configuration & Control

Video Data

Bayer

PatternGain FIU Aurora

Aurora

Output

Camera ECU

Ethernet

Pixel Pipeline Control

ESI

POD

4x ARM Cortex A53

Linux

Pixel Pipeline

ESI Unit: FPGA Firmware Overview

Confidential

Video

Input

Environment Sensor Interface Unit (Xilinx Zynq ZU9EG FPGA)

HDMI Input

Configuration & Control

Video Data

Aurora

Camera ECU

Ethernet

ESI

POD

4x ARM Cortex A53

Linux

Pixel PipelineAurora

Output

ESI Unit: FPGA Firmware Overview

Confidential

Video

Input

Environment Sensor Interface Unit (Xilinx Zynq ZU9EG FPGA)

HDMI Input

Configuration & Control

Video Data

Aurora

Camera ECU

Ethernet

ESI

PODPixel Pipeline

Aurora

Output

Video

Splitter

4x ARM Cortex A53

Linux

Radar PipelineGMSL

Output

Pixel PipelineGMSL

OutputCamera ECU

Radar ECU

GMSL

GMSL

Radar Data

Raw Data Generation for Cameras –

MotionDesk and multiple sensor models (Camera, Radar, Lidar)

Rest bus simulation

ASM

ModelDesk

…

Vehicle network

Sensor composition

combining multiple sensor

outputs incl. meta-data

ESI Unit

HDMI

Ethernet

Rear camera

Front camera

Laserscanner

Control data

Confidential

PREVIEW

PC with graphics card

and MotionDesk

Raw Data Generation for Cameras –

MotionDesk and multiple sensor models (Camera, Radar, Lidar)

Rest bus simulation

ASM

ModelDesk

…

Vehicle network

Sensor composition

combining multiple sensor

outputs incl. meta-data

ESI Unit

HDMI

Ethernet

CN+GPU

SensorSim

Rear camera

Front camera

Laserscanner

Control data

Confidential

PREVIEW

Overview Option 3 & 4 – HIL - today

29

Pro

du

ctEC

U s

pec.

pro

ject

Ethernet / IOCNet

Environment

Sensor Simulation

HDMIEthernet

Environment Sensor

Interface Unit

(FPGA)

Confidential

Netw

ork

Man

ag

em

en

t

Raw data

Target list

Object list

Sensor ECUSensor

Frontend

Application logic(Trajectory planning,

Motion control)

Object tracking

Detection, Data Proc.

Preprocessing

Environment Sensor

Interface POD

(FPGA)

Automotive

Simulation Models

(ASM, SCALEXIO CN)

High-end PC and GPU (Windows)

MotionDesk Host

ESI Unit

PODs

FPGA Firmware

35

VEOS – Realistic simulations of ADAS and automated driving functions on standard PCs

38

Sensor simulation – HIL and SIL

PREVIEW

Integrated toolchain for SIL

and HIL use cases

Many sensor technologies

are adressed

Five options to test a

sensor toolchain

Toolchain with low latency

and synchronization

Use of several GPUs is

planned

API to integrate custom

models is planned

TESTING PROCESS AND AUTONOMOUS AUTOMATION

AutomationDesk Testing and Test Tools

40

Various methods of Test Development

Signal-based testing, XML and xIL-API Open Standards

Manage all tools in the ADAS Testing Process

• fit for purpose for

developing safety related

software according to IEC

61508 and ISO 26262.

• pre-qualified for all ASILs

according to ISO 26262

Real-Time Testing

41

Standard PC

Real-time test programming via Python scripts (with use of specific RTT Libraries)

Real-Time Test management (download, start, stop, pause...)

Real-time platform or VEOS

Execution/Scheduling of Python real-time tests

Synchronization between real-time tests and

Simulink® model

Observers and the RTT Observer LibraryCoupling of BTC Embedded Specifier and dSPACE Systems

42

Requirements

(textual)

Test specification

(informal, structured)

Simulator (VEOS / HIL)

Requirements

(formalized)

EmbeddedSpecifier

RTT-Observer

Observer

• Passed/Failed

• Req-Coverage

• Textual requirements =>

Formalized requirements

=> Simulation based

formal verification

(~ISO26262)

• Permanent verification of

safety-critical

requirements with

observers, e.g. in parallel

to execution of ‚classic‘

AutomationDesk tests

Drastic increase in test

depth and coverage for

safety-critical functions

Many additional benefits

for test automation users

Observer

Download

Test Management for ADAS: Keeping Track of Testing Activities

Traceability and Coverage

Traceability from requirement to test result

and overall requirements coverage

For all types of requirements, e.g. safety,

functional, performance or robustness

requirements

Test Scenario Traceability

Which test scenario is tested by which tests?

Has a given test scenario been tested

successfully?

43

Monitor progress across multiple

test platforms and different

test tools

For tests by means of simulation

(MIL/SIL/HIL) as well as real-world

tests

Test reports, results overviews

and test evaluation

Test Stimuli Traceability

Which parameters and inputs

were used for which tests during

a test execution?

SYNECT Test Management

44

Manage all MIL/SIL/HIL Testing

1. Requirement integration and

coverage analysis - Connect to

PLM/ALM Tools

2. Off-the-shelf integration with

common test tools such as

AutomationDesk, Simulink, and

BTC EmbeddedTester®

3. Open interface to connect other

test tools or custom test solutions

4. Monitor, analyze, visualize test

results during the test process

5. Dynamic Test Parameterization –

handle ADAS model/test

functionality

45

Integrated Tool Chain for Testing by Means of Simulation

Euro NCAP AEB Use Case

• Automatically execute

Euro NCAP tests and

generate score results

• Automated

parameterization,

execution and evaluation

of Euro NCAP tests

• Example AutomationDesk

NCAP AEB Test Demo

available online

• Solutions for all NCAP

tests available as an

engineering service

Test Automation (TA) Framework

46

SYNECT Project Navigator

Test Management Project

Test Cases

Test Steps Manager

47

Test Automation (TA) Framework

PC cluster simulation with Virtual ECUs

50

Driving millions of kilometers on your PC

Testing at an early development stage

Highly scalable due to virtual ECUs

Deterministic and reproducible test execution

High test throughput through fast test execution

Simulation Cluster open to

Integrate with test generation methods

Cloud computing options

Simulation Cluster leverages

SIL tool chain in general (VEOS, xIL-API, ASM)

SYNECT Test Management

Real Time Testing

51

Cluster Test Management with dSPACE SYNECT

Parallel Testing and Execution Management

53

Plan and schedule test cases and

assign to specific cores (efficiency of

multi-core testing)

Offload test analysis from Test

resources (test resource efficiency)

Future – auto-optimization and API

for schedule/sequence customization

54

Optimizing HIL Testing Time

EXECUTION

EVALUATION

Test Case - 1

2 mins

4 mins

Offline PC

2 mins HIL Time

Iteration

Test case 1

Test case 2

Test case 3

Test case 4

. . .

Execution

Iteration

Test case 1

Test case 2

Test case 3

Test case 4

. . .

Post-Processing

Offline PC

Captured

DataResults

.mat / .mf4 1 Passed

.mat / .mf4 2 Failed

.mat / .mf4 3 Passed

.mat / .mf4 4 Passed

. . . . . .

HIL PC

Master Data ManagerSYNECT

SUMMARY

56

One Tool Chain for ADAS/AV TestingISO 26262 ready.

Prequalified for

all ASILs

dSPACE-Internal57

dSPACE - The Right Partner

for Autonomous Driving

Simulation

Exhaustive

testing

Validation

Sensors Algorithms

Proto-

typing

Virtual

test drives

58

Thanks for listening!

© Copyright 2017, dSPACE Inc.All rights reserved. Written permission is required for reproduction of all or parts of this publication. The source must be stated in any such reproduction.This publication and the contents hereof are subject to change without notice. Benchmark results are based on a specific application. Results are generally not transferrable to other applications.Brand names or product names are trademarks or registered trademarks of their respective companies or organizations.