results of model-based testing in automotive

Results of Model-Based Testing deployments performed for the Automotive

Embedded Systems Validation

MaTeLoBoost your test efficiency

AGENDA

ALL4TEC MBT History

Automotive Market

Automotive Electronic Challenge

Automotive Electronic Testing

ALL4TEC Customer Slide

Why MBT in Automotive

Usage Model Design

Test Case Generation

Test Execution

Test Campaign Analysis

Results


MaTeLo History

10 years of MBT

Starting in Automotive with Magneti Marelli

MaTeLo as “Markov Test Logic”

1

•Model with INI File

2 •Model with

PowerPoint

3 •Test Campaign

Analysis

4 •Test Automation

•NI TestStand

•EXAM

•Proprietary

•Requirements

• IBM Doors

•XML, CSV

•Test Management

•HP QC

•Test Oracles

• Simulink

• Scilab

•Python

5 • Eclipse

ALL4TEC MBT Experience in Automotive

Instruments Clusters

Multimedia Unit

Seat Belt ECU

Display

Body Control Unit

Engine Control Unit

Chassis ECU

Climate Control

Stop & Start

Electrical Lateral Door

….

intelligent-vehicles.

Automotive Electronic

N Lines of Codes

Ni.com Software Engineering

Economics

10 M

40 %

*1000

80 M

The Simple Automotive World

Standard

Tools

Safety

Product Lines Platform

Car Variant

Complexity

Process

CMMI Spice

AutoSar

HIS

Vector

ETAS

Mathworks

dSPACE NI

Menthor

OSEK

MISRA

UDS

ASAM

ISO26262 Multi OEM

ECU ESP

VW, AUDI, PORSCHE,

SKODA, SEAT,

GM, CHEVROLEY, OPEL,

PEUGEOT, CITROEN

A Heavy Truck

is almost unique

EMS

NAVIGATION

1-10 Networks

20-110 ECUs

x0 000 signals x km of wire

x00 000

Requirements

ECU 1

An Integration Market !

ECU A ECU 1

ECU 4

ECU 2

ECU 3

ECU C

ECU B

OEM 1 OEM 2

T1 A

COMPETITION

Parts Suppliers

Vehicle Manufacturers

OEM/T1 Integration V Cycle

Parts Suppliers

System Definition

Architecture Design

ECU Requirements

System Design

System Validation

ECU Verification

System Verification

Product Validation

Software Development

INTEGRATION


Test Level ?

Parts Suppliers

System Definition

Architecture Design

ECU Requirements

Architecture Design

System Validation

ECU Verification

System Verification

Product Validation


System MIL

Feature SIL

Feature MIL

Feature PIL

ECU HIL

ECU HIL

System HIL

Field


Test Object ?

Parts Suppliers

System Definition

Architecture Design

ECU Requirements

Architecture Design

System Validation

ECU Verification

System Verification

Product Validation


System MIL

Feature SIL

Feature MIL

Feature PIL

ECU HIL

ECU HIL

System HIL

Field

Multi Domain Model

Software Model

Software

Embedded

Code

ECU

Electronic

System

Vehicule


The Testing Spectrum

Embedded Software Control Logic

Software Level Testing

Control Procedure Level Testing

Network Level Testing

Diagnostic Level Testing

Automobile Functional Integration HIL

ATZ Magazine

Journée Utilisateurs MaTeLo 2012

> 100 peoples

Why MBT (1)

Why MBT (2)

MBT in the V Cycle

MBT in the Agile Cycle (Continuous Integration)

MATELO USAGE MODEL

(4) CONSISTENCY

TESTS <-> REQUIREMENTS

BASED ON USE CASES

(2) ALL STRATEGIC

USAGES SCENARIO

(3) FORMAL

REQUIREMENT

TEST SPECIFICATION

Enabler for

Global Test Strategy

Graphical Design/Review

Automatic Generation

Model Checking

Impact Analysis

(1) BIG PICTURE

MATELO USAGE MODELS

BEGINNING

END

STATE

TRANSITION

MACRO CHAIN

Setup “Transition” as “Test Step”

Mapping of Requirements

Configuration for Test Automation

Usage Model Design

REQUIREMENTS MANAGEMENT

MaTeLo Requirements Library MaTeLo Usage Model

Model & Trace (N,M)

IBM Rational Doors

Import

Requirement Update New Baseline

Update Quick Model Update

Requirements Change Analysis

Model Impact

Highlighting

Stimulations

Inputs Stimulation

Equivalence Classes

Timing

Verifications Points

Expected Ouputs Results

Timing

MODEL TRANSITION = TEST STEP

Requirements Test Automation

Select_Gear(5)

Accelerate(100%)

Check_Speed(185)

Computed Test Oracle

Outputs = f(Inputs)

Derived Test Oracle

MATELO PROFILE

Every Alternative can be qualified by Weight

Test Profiles

Operational Profiles

347

0-100-150-200-500

87 (+/- 25)

Data distribution Path frequency

MATELO PROFILE

0-500 0-500 0-500

MATELO PROFILE

Profile 1

87 (+/- 25)

MATELO PROFILE

0-100-150-200-500 87 (+/- 25) 347

Profile 2

Automotive Product Lines

TFT

KM/H M/H

DOT MATRIX

Display

Country

Navigation

OEM

Required

Feature Products Line

Constraint

Variability Point

Product

MaTeLo Product Line Manager

TCG

TCG

TCG

TCG

MG

Product Usage Model

Derived from PL Model

Test Cases Generation

By Product Usage Model

THEORETICAL MaTeLo WORKFLOW

Automation

Test Framework

Test Scheduler

Simulator

System

Under

Test

Test

Execution

Log

Usage

Model Test

Cases

Functional

Requirements

Atomic

Test

Action

Test

Cases

Verdicts

Confidence

& Coverage

Reports

Test

Scripts

Automatic Test Framework Setup

Test Automation Framework integration

MaTeLo Eco-System

TCL Proprietary Script

EXAM

TestStand - VeriStand

Java

CAPL

Python

VB

HTML

ATX

MaTeLo

Select the Test Cases

4 Generation Strategies

depending on your Test Strategy

4 TESTS STRATEGIES

Most probableapproach

Start

Close

FREQUENCYFOCUS

Start

Close

Risk basedapproach

CustomTest profile

CRITICALITY, COMPLEXITYUPDATE FOCUS

Arcs coverageapproach

Close

Start

REQUIREMENTSCOVERAGE

UsageTest profile

Usageapproach

Start

Close

OPERATIONALCOVERAGE

New Test

Cases

Generation

TEST CAMPAIGN PROCESS

Test Cases

SystemRequirements

UsageModel

Test ExecutionPlatform

SystemUnderTest

Adaptor

Test Scripts

Requirements Traceability

Model Coverage

Reliability

Mean TimeTo Failure

Intensityof Test

Test CasesTest

Cases

SystemRequirements

SystemRequirements

UsageModelUsageModel



SystemUnderTest

SystemUnderTest

Adaptor

Test Scripts

AdaptorAdaptor

Test ScriptsTest

Scripts


Model Coverage


Model Coverage

Reliability

Mean TimeTo Failure

Intensityof Test

Reliability

Mean TimeTo Failure

Intensityof Test

Test Cases

SystemRequirements

UsageModel


SystemUnderTest

Adaptor

Test Scripts


Model Coverage

Reliability

Mean TimeTo Failure

Intensityof Test

Test CasesTest

Cases

SystemRequirements

SystemRequirements

UsageModelUsageModel



SystemUnderTest

SystemUnderTest

Adaptor

Test Scripts

AdaptorAdaptor

Test ScriptsTest

Scripts


Model Coverage


Model Coverage

Reliability

Mean TimeTo Failure

Intensityof Test

Reliability

Mean TimeTo Failure

Intensityof Test

Test

Cases

Syste

mRequirem

ents

Usage

Model

Test

Execution

Pla

tform

Syste

mU

nder

Test

Adapto

r

Test

Scripts

Requirem

ents

Tra

ceability

Model

Covera

ge

Reliability

Mean T

ime

To F

ailure

Inte

nsity

of

Test

Test

Cases

Test

Cases

Syste

mRequirem

ents

Syste

mRequirem

ents

Usage

Model

Usage

Model

Test

Execution

Pla

tform

Test

Execution

Pla

tform

Syste

mU

nder

Test

Syste

mU

nder

Test

Adapto

r

Test

Scripts

Adapto

rAdapto

r

Test

Scripts

Test

Scripts

Requirem

ents

Tra

ceability

Model

Covera

ge

Requirem

ents

Tra

ceability

Model

Covera

ge

Reliability

Mean T

ime

To F

ailure

Inte

nsity

of

Test

Reliability

Mean T

ime

To F

ailure

Inte

nsity

of

Test

Stopping Quality Criteria

Coverage

Reliability

Confidence Indicators

Test

Cases

Syste

mRequirem

ents

Usage

Model

Test

Execution

Pla

tform

Syste

mU

nder

Test

Adapto

r

Test

Scripts

Requirem

ents

Tra

ceability

Model

Covera

ge

Reliability

Mean T

ime

To F

ailure

Inte

nsity

of

Test

Test

Cases

Test

Cases

Syste

mRequirem

ents

Syste

mRequirem

ents

Usage

Model

Usage

Model

Test

Execution

Pla

tform

Test

Execution

Pla

tform

Syste

mU

nder

Test

Syste

mU

nder

Test

Adapto

r

Test

Scripts

Adapto

rAdapto

r

Test

Scripts

Test

Scripts

Requirem

ents

Tra

ceability

Model

Covera

ge

Requirem

ents

Tra

ceability

Model

Covera

ge

Reliability

Mean T

ime

To F

ailure

Inte

nsity

of

Test

Reliability

Mean T

ime

To F

ailure

Inte

nsity

of

Test

Bugs New

Package

Test

Cases

Syste

mRequirem

ents

Usage

Model

Test

Execution

Pla

tform

Syste

mU

nder

Test

Adapto

r

Test

Scripts

Requirem

ents

Tra

ceability

Model

Covera

ge

Reliability

Mean T

ime

To F

ailure

Inte

nsity

of

Test

Test

Cases

Test

Cases

Syste

mRequirem

ents

Syste

mRequirem

ents

Usage

Model

Usage

Model

Test

Execution

Pla

tform

Test

Execution

Pla

tform

Syste

mU

nder

Test

Syste

mU

nder

Test

Adapto

r

Test

Scripts

Adapto

rAdapto

r

Test

Scripts

Test

Scripts

Requirem

ents

Tra

ceability

Model

Covera

ge

Requirem

ents

Tra

ceability

Model

Covera

ge

Reliability

Mean T

ime

To F

ailure

Inte

nsity

of

Test

Reliability

Mean T

ime

To F

ailure

Inte

nsity

of

Test

New version

Test

Cases

Syste

mRequirem

ents

Usage

Model

Test

Execution

Pla

tform

Syste

mU

nder

Test

Adapto

r

Test

Scripts

Requirem

ents

Tra

ceability

Model

Covera

ge

Reliability

Mean T

ime

To F

ailure

Inte

nsity

of

Test

Test

Cases

Test

Cases

Syste

mRequirem

ents

Syste

mRequirem

ents

Usage

Model

Usage

Model

Test

Execution

Pla

tform

Test

Execution

Pla

tform

Syste

mU

nder

Test

Syste

mU

nder

Test

Adapto

r

Test

Scripts

Adapto

rAdapto

r

Test

Scripts

Test

Scripts

Requirem

ents

Tra

ceability

Model

Covera

ge

Requirem

ents

Tra

ceability

Model

Covera

ge

Reliability

Mean T

ime

To F

ailure

Inte

nsity

of

Test

Reliability

Mean T

ime

To F

ailure

Inte

nsity

of

Test

Requirements

Coverage

Functional Coverage

Operational

Coverage

Stopping criteria

Test Execution

Test Campaign Analysis

Results with a Powertrain Parts Supplier

Results with a Car Manufacturer

Results with a Body Supplier

Results with a Car Manufacturer

QUESTION ?
