simulation software-in-the-loop · osi or oem-test results in mat or hdf5 concr. openscenario...

www.pegasusprojekt.de

TESTING AND SAFEGUARDING – Booth No. 21

SIMULATION –

SOFTWARE-IN-THE-LOOP

PEGASUS Method for Testing HIGHLY Automated Driving Functions

PEGASUS enables the connection to existing tools.

How are existing SiL-Tools integrated into

the toolchain and the test field?

Simulation Proving

Ground

Real

World Test

Test coverage >> 10,000 simulation runs per scenario, especially critical scenarios

Effort for qualification test equipment /

tools / methods

High effort for simulation model parametrization und validation

(driving dynamics, sensor, actuators, traffic agents) as well as

overall verification of the simulation chain with real driving data.

Effort for test setup / preparation Low, but depending on the degree of test automation

Effort for test execution Low, "only" computing power necessary

Standardization methodology / tools

Methodology given by Pegasus, but no, or only conditional,

standardization of manufacturer-specific simulation models and

tools

Current limits Ego vehicle, environment and sensors not yet fully modeled in the

model

Goal: Test of all scenarios from the database and their variations to

identify collision-relevant scenario characteristics for the automated

system:

- Number of test cases: „BIG“ >> ~ 10,000

Input: Scenarios and their parameters (including distributions), Pass-

criteria / criticality metrics, original ECU code

Output: Scenarios assessed against pass criteria and, if applicable,

their probability of failure (in the case of accidents)

B

O

O

T

H

2

3

B

O

O

T

H

2

7

The objective of PEGASUS Testing was to

develop the PEGASUS method regarding

completeness, correctness and consistency.

The module SIMULATION is an enabling

element / tool.



SIMULATION –


PEGASUS enables the connection to existing tools. How are existing SiL-Tools integrated into the toolchain and the test field?

Motivation:

• Using a common toolchain to execute tests

• Common interfaces inside the toolchain enables an interchangeability of tools

• Using in PEGASUS developed Sensor- Models and Stochastics- Module

Common Toolchain

Test Automation

Test-Case Generation

Stochastics-Module

Test Control

Scenario-

database

Simulationstool

Environment Model

Sensor

Models

Overall

Result

Result

Evaluation

OSI or OEM-

Connection

Transpiler OpenDrive-

Generator

Parametrization

AD

Function

Vehicle Model

Logical Scenario

Logical „Simplified

Road“

+ Parameter description

file

+ Evaluation criteria

OSI or OEM-

Connection

Test Results in

mat or hdf5

Concr.

OpenScenario

Concr. OpenDrive


Visualization of Results

Simulation Run

Min

TT

C (

Obje

ctive F

unction)


SIMULATION –


Test-Case Description

Pass-Criterium: no accident, i.e.

time-to-collision > 0 s

Varied Parameters:

Video

Simulation-Tool DB

1 logical scenario

+ parameters

+ test criteria

Functions + Models

Sim-

Results:

JSON +

hdf5 „1000“ concrete

scenarios

Results Evaluation (local)

OSI

Sensor Models

Overall-Results

Test Automation

Stochastics-Module

BMW - Realization

Critical and Uncritical Scene

⇒ 𝑇𝑇𝐶min= 0.83 s

Critical Scene:

L4_Ego_Speed_Initial 31.5 m/s

L4_Obj_Position_Initial 150.3 m

L4_Obj_Speed_Final 14.9 m/s

L4_Obj_Speed_Initial 30.3 m/s

Convergence Behavior of Optimizer

Ego

Parameter Lower

Bound

Upper

Bound

L4_Ego_Speed_Initial 20 m/s 36 m/s

L4_Obj_Position_Initial 120 m 160 m

L4_Obj_Speed_Final 12 m/s 33 m/s

L4_Obj_Speed_Initial 12 m/s 33 m/s

L1_Road_Length 3000 m 4000 m



SIMULATION –


-Realization

Simulation-Tool DB

1 logical scenario

+ parameters

+ test criteria

Functions + Models

Sim-

Results:

JSON +

mat or


scenarios


OEM

Sensor Models

Overall-Results

Test Automation

Stochastics-Module

Test-Case Description

Cut out vehicle standing

vehicle

Pass-Criterium:

No accident:

Distance >0 m EGO

Result:

Varied Parameters :

TTC (max: green, min: red)

50

57.5

65

72.5

80

0.6 0.7 0.8 0.9 1.0

𝜇

𝑣 [

km

/h]

1 2 Uncritical case Relevant case

Video 1 Video 2

1

2

Parameters of uncritical case:

𝑚 = 1600 kg; 𝑠 = 5 s; 𝑑𝐴 = 20 m; 𝑣 = 50 km/h;

𝑇𝑇𝐶min = 1.892 s

Parameters of relevant case:

𝑚 = 2200 kg; 𝑠 = 2 s; 𝑑𝐴 = 0 m; 𝑣 = 80 km/h;

𝑇𝑇𝐶min = 0.676 s

Parameter Lower

Bound

Upper

Bound

Mass EGO 𝑚 1600 kg 2200 kg

Cut-out duration 𝑠 2 s 5 s

Cut-out waypoint 𝑑𝐴 0 m 20 m

Velocity 𝑣 50 km/h 80 km/h

Friction value 𝜇 0.6 1.0

Simulations No

accident accident

„relevant“

(TTC < 1,0 s)

3125 2759 366 1725



SIMULATION –


-Realization

Simulation-Tool DB

1 logical scenario

+ parameters

+ test criteria

Functions + Models

Sim-

Results:

JSON +

mat or

hdf5

Testnachsteuerung

„10000“ concrete

scenarios


OEM

Sensor Models

Overall-Results

Test Automation

Stochastics-Module

2 Relevant Case

Time [s]

TT

C [

s]

𝑣 [

km

/h],

𝑎𝑉

𝑒𝑟

𝑧 [

dm

/s²]

Detail plot relevant case

Proving ground Simulation

Matching proving ground/simulation:

• Situation passed

uncritical Case



SIMULATION –


-Realization

Simulation-Tool DB

1 logical scenario

+ parameters

+ test criteria

Functions + Models

Sim-

Results:

JSON +

mat or


scenarios


OEM

Sensor Models

Overall-Results

Test Automation

Stochastics-Module

Quality criterion regarding the probability of failure

• Standard deviation of failure probability

• Confidence interval

Simulations

7x10-6

Pro

ba

bili

ty o

f fa

ilure

<10% Sta

nd

ard

de

via

tio

n

Stepwise approach of the failure

region until a specific safety limit

has been reached

TTC < 1.0: pf = 0.558906

TTC < 0.5: pf = 0.210416

TTC < 0.4: pf = 0.014846

TTC < 0.3: pf = 0.000061

TTC < 0.2: pf = 0.000024

TTC < 0.1: pf = 0.000014

TTC < 0.0: pf = 0.000007

Determination of the probability of failure by using exemplary data and distributions

𝜇

𝜇

𝜇

𝑣 [km/h]

𝑣 [km/h] 𝑣 [km/h]

Characterization of the parameter

space

Distribution of cumulated

probability

Distribution of cumulated

probability of violating criteria

probability probability TTC_Vcoll

0.6 0.7

0.8

0.9 1.0

0.6

0.7

0.8

0.9

1.0 0.6

0.7

0.8 0.9 1.0

50 57.5

65 72.5

80

50 57.5 65 72.5 80

50 57.5

65 72.5

80

0.25

0.15

0.05

0.000006

0.000004

0.000002

2.0

-4

-2

0



SIMULATION –


-Realization

Simulation-Tool DB

1 logical scenario

+ parameters

+ test criteria

Functions + Models

Sim-

Results:

JSON +

hdf5

Testnachsteuerung

„10000“ concrete

scenarios


OEM

Sensor Models

Overall-Results

Test Automation

Stochastics-Module


Examples of Critical and Uncritical Cases

Pass-Criterium: no collision, i.e. RSS < 1

2

Ego SCT

1

Ego Lane Change

Varied Parameters Lower

Bound

Upper

Bound Dep.

Distance to Ego d1 -50 m 50 m -

Distance to Ego d2 -50 m 50 m d1 > d2

Distance to Ego dSCT 100 m 200 m -

Brake Trigger Dist. dSctTrig. 40 m 100 m -

Brake Accel. aSCT 1 m/s² 10 m/s² -

Optimization Generations Simulations

Maximize: Fitness(RSS) 30 1500

Critical Case d1 = 30 m d2 = 1 m Fit 𝑅𝑆𝑆 = 100

Uncritical Case d1 = 30 m d2 = -30 m Fit 𝑅𝑆𝑆 = 0



SIMULATION –


-Realization

Simulation-Tool DB

1 logical scenario

+ parameters

+ test criteria

Functions + Models

Simulation-

data „10000“ concrete

scenarios


OSI/

OEM

Continental Camera Model

Overall-Results

Test Automation

Stochastics-Module

Optimization towards a critical scenario

Pass-Criterium: no accident, i.e.

time-to-brake > 0 s

Varied Parameters:

Cut-in from right

Parameters Lower

Bound

Upper

Bound

Object position (initial) 0 m 260 m

Object speed (initial) 5 m/s 48 m/s

Object speed (final) 4 m/s 50 m/s

Ego speed (initial) 18 m/s 36 m/s

cut-in duration 𝑡 2 s 9,4 s

Ego

In several iterations, the particle swarm optimizer

aims to optimize the parameters of the scenario to

generate a critical one.




SIMULATION –


-Realization

Simulation-Tool DB

1 logical scenario

+ parameters

+ test criteria

Functions + Models

Simulation-

data „10000“ concrete

scenarios


OSI/

OEM

Continental Camera Model

Overall-Results

Test Automation

Stochastics-Module


simulation software-in-the-loop · osi or oem-test results in mat or hdf5 concr. openscenario...

Documents