artemis project mbat: advanced validation & verification of embedded systems af brian nielsen,...

© MBAT, ARTEMIS project 269335ARTEMIS Joint UndertakingARTEMIS Joint UndertakingThe public private partnership for R&D actors in embedded systems

http://www.mbat-artemis.eu/

© MBAT, ARTEMIS project 269335ARTEMIS Joint UndertakingARTEMIS Joint Undertaking

© Daimler

Embedded Systems in a Car (ECUs) as example Targets of MBAT

3

3‐4 networks100+ ECUs2017 autonomous2020 driverless

© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking

© Daimler © Daimler

Example application domain for MBAT:automotive HIL integration test environment for model-based testing of embedded systems (interactions)

Automotive Test Environment

4


MBAT’s overall Challenges

V&V technologies are still not effective and efficient enough

V&V costs for Embedded Systems are too high (still up to 50% of Embedded System’s total development costs)

V&V technologies should improve the error detection rate

HIL Test Environment © Daimler

5


MBAT‘s European impact

MBAT outcomes will contribute to increase the competitiveness of European transportation products industry

MBAT will support higher quality European transportation products at reduced development costs

6


MBAT’s Project Character

MBAT is an ARTEMIS project, thus focussing on embedded

systems an industrial-oriented R&D project to transfer tool

innovations and academic research into industrial application

strongly driven and evaluated by industrial use cases

7


MBAT’s Market Impact

MBAT will increase the competitiveness of European key players in transportation domain by reducing V&V costs for embedded systems by at least 20 % (keeping the

planned level of quality) shortening time‐to‐market by at least 20 % increasing the coverage of the embedded system under V&V by at least 30

% significantly increasing the probablitity to uncover errors enabling higher quality and safer embedded systems & embedded

systems based products

8


MBAT Technological Innovation

MBAT = Combined Model‐based Static Analysis and Dynamic Testing of Embedded Systems

Test & Analysis Models

Test Cases

Analysis Results

Test Results

Analysis Cases

Embedded Systems Descriptions

Dynamic Tests

Static Analysis

Test

Analysis

10


MBAT Outcomes

Industrial‐approved MBAT Reference Technology Platform (MBAT RTP) supporting Validation & Verification of Embedded Systems

Experience Packages describing the usage of the RTP in industrial domains (automotive, aerospace, rail)

11


UC No. Use case name Use case driverAutomotive Use Cases

UC A1 Brake-by-Wire VOLVOUC A2 Common Powertrain Control (CPC) DAIUC A3 Adaptive Brake Light (ABL) DAIUC A4 Turn Indicator Control (TIC) DAIUC A5 Transmission Controller Product Line RICUC A6 Passive Balancing AVLUC A7 Hybrid Power Train Control Unit AVLUC A8 Virtual Prototype Airbag ECU IFAT

Aerospace Use CasesUC AE1 Flight Control Program AIRUC AE2 ACSL Component for Flight Control Computer AIRUC AE3 Flight Warning Program AIRUC AE4 Flight Management System/UAV ASIA

UC AE5 Degraded Vision Landing Aid System DeViLASystem EADS DE

UC AE6 TALARION - Unmanned Aerial Vehicle (UAV) EADS IWUC AE7 Flight Guidance System (FGS) RCFUC AE8 Attitude and Altitude (A&A) for Helicopters RCFUC AE9 Spacecraft Central Software-Sentinel 3 TAS

Rail Use CasesUC T1 Automatic Train Control ALSTOMUC T2 Rapid Transit Metro System (Ansaldo STS) ANSALDOUC T3 Validator of the ZLB ATOP System SIEMENS

MBAT Use Cases

12


(Automated) V&V

Techniques

Dynamic Techniques

Static Techniques

Testing Monitor‐ing Simulation Theorem

Proving

Symbolic Exec

Model‐checking

Abstract Interpreta‐

tionRefinement‐checking

MiL TestingStatistical Model

Checking

Runtime Verifi‐cation

Software Model‐checking

(hybrids)

Classification of Techniques


Req. VerificationPlan/Status T&A Model(s)

Analysis Cases

Test Cases

Analysis

Test

CoverageResults,

Main MBAT Method


Correct code, or verify weaker invariant, and analyse implicationon models level

Verify req using sim/test as approximation,or re-verify using strengthenassumption (or refine req.)

V&

V P

lanning

Success(verified)

Failed Reqs

Inconclusive

Suspectedor new case

Correct model, design, code, and repeat V&V of all impactedsys. Add cases for regression check

Success(pass)

Failed Reqs

Inconclusive


Correct system, and analyze model in context of trace and test case to ruleout similar errors

V&V Objectivesto be analysed

V&V Objectivesto be codechecked

Success(verified

Failed Reqs

Inconclusive


”Maybe satisfied” property: derivetest High warning densityHigh complexity

Define analysis cases for model-analysisDefine invariants for static codeanalysisUncovered items: try to target theseusing model analysis

1) Make initial V&V Plan that maprequirements/V&V objectives to most suitable techique

5) Update V&V plan and status based on results

Define new analysis or test cases for model-analysis (or invariants for static codeanalysis)

1 2

3

3

3

2) Construct analysisand/or test model(s)

3) Execute4) evaluate results

Model-Analysis

(MB) Testing

Code-Analysis

use testing

Engineering Artifacts

V&V Objectivesto be tested

4 Feedback

T&A

mod

el(s

)4 Feedback

55

Refine req and test case

ok

ok

ok


Hybrid powertrain control unit (HCU) that is responsible for coordinating the energy flows between engine, electrical motor, and the battery.

AVL’s HCU Initial Combination


Refinement ProcessOverall Methodology

Workflow / Combination Patterns

(sub) MethodInstance

RTP Instance

• Framework describing general workflow and most A&T combination strategies

• Holistic view• Domain‐ and tool‐independent

• Pattern=reusable solution to a commonly occurring problem

• Pattern for common A&T Combination strategies• Typically focuses on only a part of the V&V flow

• A specific chosen set of notations (reqs, models, traces, etc.)

• Specific type of results and data to be exchanged (syntax and semantics)

• Specific set of tools

• Workflow, and data exchange supported by the RTP• Tools integrated/interoperable


Reduce warnings from Static Code Analysis

Static Code Analyzer

Model-Analyzer

Code +config

Model generator

report

• Program slice

• Path precondition

warnings

Report merger

Confirmed defects Remaining warnings

Confirmed defects

UC T3 SIE “ZLB ATOP System”: SAT‐solving using RTT+UC A2 DAI “CPC”: model‐checking

• Semantic preserving Model

• Property

1

32

4

Purpose: Reduce number of warnings from static code analysis by more exact analysis


Instance (Work in progress)

Code +config

report

Report merger


Confirmed defects

Purpose: Reduce number of warnings from static code analysis by more exact analysisPre-condition: first step conducted by abstract interpretation (over-approximation)Maturity: researchVariants:Notes:

Astrée+slicer+exchange format for invariants

Uppaal

Static Code Analyzer Model generator Model Analyzer

WarningsPreconditionSlice

Significant effort!!UC A2 DAI “CPC”: model‐checking


Reduce warnings from Static Code Analysis

Static Code Analyzer

Test Execution

Code +config

Test input generator

report

• Program slice

• Path precondition

warnings

Report merger


Confirmed defects

• Instrumented Program (Oracle)

• Test case

UC AE8 RWC “Attitude and Altitude for Helicopters”

1

32

4

UC T3 / SIE “ZLB ATOP System”


Increase Coverage by Analysis

Simulation based test generator

Path synthesizer

Test suite

Executor + Coverage Evaluator

Test Model

Coverage Analyzer

SUT

Cover-age report

Test Case

Test Input

Model-checker

Cover-age report

(Counter examplebased) Test Case

Two Patterns? • Model‐coverage• White‐box SUT/Code

Coverage

Same test suite / test format?Is it possible to transfer a path synthesized test case to model level?

Can model serve as Oracle?

Alternative: Coverage based test generation+ Coverage completion by simulation

oracle

UC_AE6 EADS TALARION UC_AE7 RWC “Flight Guidance System”: MC/DC CoverageUC T2 ? ASTS 3.1.9 ANSALDO “Rapid Transit Metro System”

1

3

2

4 5

6


MBT with analysis 1

Model-based test generator

Test suite

Executor

Test Model

SUT

Model-checker

Test objectives

(Formalized) Requirements

Fail: Hypothesis: most likely impl is wrong because model was checked wrt req’s Pass: Hypothesis: Impl satisfies requirements because model satisfy reqs and impl refines model

Analysis Objectives

Report

e.g. UC A1 Volvo BBW

1

3

2

4 5


MBT with analysis 2


Test suite

Executor

Analysis Model(Env + SUT)

SUT

Model-checker

Test objectives


1. Model‐check could not verify all requirements on analysis model (spate space too large)

2. Model‐check in context (as environment model/input) of failing test case (to reduce state space)

3. Higher confidence, targeted analysis: confirm/exclude “similar” errors

Analysis Objectives

Report

Abstracted failed test trace (or observed suspect behavior)

13

2

Purpose: Rule out further defects along known failing test


Target MBT to failing test case


Test suite

Executor

Test Model

SUT

Model-checker

Test objectives


neighborhood• A related test path found by

choosing alternative outcome at branching point in the original path[Peled, in FME 2001]

• Small “trace distance”

Analysis Objectives

Report

failed testcase

1

3

2

Purpose: Generate additional related test cases in the same model neighborhood due to bug cluster assumptionPre-condition: failed test case, notion of neighborhood


Target MBT to suspect areas


Test Cases

Static analysisSU V&V

Model Mapping: links code level element (“defect area” e.g., function/statement) to Model‐level element (eg. component or transition)• Traceability info? • Auto generated code (e.g. Daimler impact

analysis for Simulink)• Manual inspection

Report

Test Objectives

Analysis Objectives

Model

High warning densityHigh (cyclomatic) complexity

Model Mapping

Model-check

1

3b

3a

2

Purpose: Target suspected parts of SUV&V with additional analysis and test cases Pre-condition: notion of neighborhood, mapping


MBAT RTP in more Detail

A Reference Technology Platform (RTP), like the ARTEMIS MBAT RTP, provides a set of management or engineering methods and processes, as well as engineering tools, which will be used to compose/build a complete engineering environment

Integrated subset of RTP components. The interoperability

approach is based on the IOS

(RTP – Tailoring)

An Interoperability Specification (IOS) will guarantee these needs for interoperability and collaboration between tools across the entire engineering lifecycle

30

artemis project mbat: advanced validation & verification of embedded systems af brian nielsen,...

Technology

mbatusecases uc

vv costs

resultsdefine analysis

repeat vv

initial vv plan

update vv plan

tested5vv objectives

requirementsvv objectives