rachid kherrazi-testing-asd-interface-compliance-with-asd spec

Rachid Kherrazi 8-10-2014 Amsterdam

TESTING ASD INTERFACE COMPLIANCE WITH ASDSPEC “A TOOL COMBINING THE BEST OF TWO TECHNIQUES”

PEOPLE INVOLVED

• Rachid Kherrazi

– Sr. Consultant-Project Leader @ Nspyre

– Domain

• Process and Product Improvement

• Model Based Testing and Model Driven Engineering

• iSQI Accredited Certified Model Based Tester Trainer

• Also : Arjan van Der Meer & Marc Hamilton

2

ABOUT NSPYRE

MARKET SEGMENTS

• High Tech

• Traffic & Infra

• Industry

AREA’S OF EXCELLENCE

(Model Based) Systems Engineering /Model Driven

Engineering / Model Based Testing /Industrial Automation /

Simulation / Big Data / Mobile Solutions

3

CONTENTS

1) Introduction to Model Based Testing

2) Generation of test cases form ASD interface model using

ASDSpec tool

3) Case Study and Results

4) ASDSpec Technology

4

V-MODEL AND MODEL DRIVEN SYSTEM DEVELOPMENT

Requirements

Analysis

System Functional

Analysis

Design Synthesis

SW Analysis &

Design

SW Implementation & Unit

Test

Module Integration

& Test

(Sub-)System

Integration &Test

Model/

Requir

em

ents

Reposi

tory

Model Based System

Engineering

System Acceptance

Some of related technology

5


Requirements

Analysis

System Functional

Analysis

Design Synthesis

SW Analysis &

Design


Test

Module Integration

& Test

(Sub-)System

Integration &Test

Model/

Requir

em

ents

Reposi

tory

Model Driven

Engineering

System Acceptance


6


Requirements

Analysis

System Functional

Analysis

Design Synthesis

SW Analysis &

Design


Test

Module Integration

& Test

(Sub-)System

Integration &Test

Model/

Requir

em

ents

Reposi

tory

System Acceptance

Model Based Testing


7

Integration Design

Coding

Requirements System test

Test case specification

(design of logical test case)

(reduction of number of test cases by application of test techniques

Test execution

(reporting)

Test case generation

(design of physical test case)

(selection of input values and calculation of expected results)

PAGE 8

TEST PROCESS

3 main steps in test process

Module

Subsystem

System

8

MBT IS THE AUTOMATION OF TEST CASE GENERATION

Manual Automatic

Test specification

Test execution

Test generation

Model

base

d t

est

ing

Tra

dit

ional

manual

test

ing

Tra

dit

ional

auto

mati

c t

est

ing

Manual

Scripting

Testing skills + Scripting skills + Modeling skills

Manual

Modeling

9

MBT PROCESS IN A NUTSHELL

Requirements

SUT

Expected outputs

(Test Oracle)

Inputs

(Test sequences)

Verdict Failed Actual result # expected

result

Test suite

generate

author

feedback

feedback

feedback

Feedback

(via review, model

exploration, Model

checking)

control

Test Model

Actual outputs

Implementation Bug

Modeling error

Specification error

Design error

Conformance failures

10

EXAMPLE: MODELING WITH SPEC EXPLORER LOOK AND FEEL

Explore &

Analyze

Remodel

Generate

C# Model (or other .Net

Language)

Model Graph

Test Suite

11

Execute

Nunit

Note: this is not implementation

code, this is the textual Test model

(describing the interface behavior)

System Under Test

WHY ARE MODELS NEEDED FOR TESTING?

12

Component A

Component C

Component B

Unit A1

Unit A2

Unit B1 Unit B2

Unit C1 Unit C2

Unit testing is insufficient

- Single unit may work properly

in isolation

- Incorrect interaction between

units may cause serious

security/reliability failures

System-level testing:

- Requires model of externally visible system behavior

- Traditional FSM-based testing does not scale

- Behavior is often reactive/nondeterministic

- Implementation is multi-threaded or distributed

- Thread scheduling hard to control

- State space is typically infinite

SOME BENEFITS OF MODEL BASED

• Increased productivity (increased automation)

• Better test script maintenance

• Improved product reliability (early defect detection)

• Agility ( Easily react to new feature changes, Reusability of test semantics, Early test engagement, Drive quality upstream)

• Increased employee satisfaction (challenging, new horizon, fun)

13

MBT: CHALLENGES AND OPPORTUNITIES

Challenges: MBT is not easy: Different mind shift from traditional testing (Modeling skills)

and steep learning curve and high ramp up cost (tooling)

Opportunities: Modeling skills specialized training and certification

e.g. iSQI Certified MBT training (CMBT)

Nspyre is a iSQI training provider

Tooling More (Open Source) supporting integration of modeling, M2M

transformation,…

e.g. Nspyre ASDspec

14

SUPPORTING TOOLING

Manual Automatic

Test specification

Test execution

Test generation

Model

base

d t

est

ing

Tra

dit

ional

manual

test

ing

15

Testing skills

Manual

Modeling

MBT n

ext

Semi-automatic

Modeling

Manual

Scripting

+ Scripting skills Modeling skills

Tra

dit

ional

auto

mati

c t

est

ing

IDEA: REUSE EXISTING MODELS

16

ASD Interface Model ASDSpec

SpecExplorer

VERUM ASD:SUITE

17

Design model

Interface model

Requirements

Features

Use cases

Prototype

Decomposition

Code generation

Simulation & verification

NSPYRE ASDSPEC

18

Design model

Interface model

Requirements

Features

Use cases

Prototype

Decomposition

Test model

Test cases

ASDspec

Code generation

Simulation & verification

Testing

Note: reuse only the

Interface model! Design

model is not used

WHAT IS VERIFIED BY ASD

19

System Under Test

Component A

(ASD genereted)

Component C

(ASD genereted)

Component B

(hand written)

Unit A1

Unit A2

Unit B1 Unit B2

Unit C1 Unit C2

ASD verification doesn’t cover:

• Handwritten /legacy code

• Interaction between generated/handwritten component

• Data interaction

With MBT (Spec Explorer) we can cover all the above aspects!

ASDSPEC: WORKFLOW

Spec

Explorer

Model

Cord

file

Spec

Explorer

Test

Cases

20

tester /modeler

Cord file:

• Test Selection /Slicing /Scenario Selection

• Data Combination

• Model Composition

Model:

• Interface behavior

Spec Explorer (MBT tool):

• Generate automatically test cases

MBT

ASDSPEC: WORKFLOW

ASDspec

ASD

Interface

Model

Spec

Explorer

Model

Cord

Script

Spec

Explorer

Test

Cases

Reuse existing ASD interface models to generate Spec

Explorer MBT models automatically

21

MBT

ASDSPEC: DATA AND MODEL COMPOSITION

ASDspec

ASD

Interface

Model

Spec

Explorer

Model

Cord

file

Spec

Explorer

Test

Cases

Generated model can be extended with any Spec Explorer feature

in cord file (config file)

22

Cord file:

• Test Selection /Slicing /Scenario Selection

• Data Combination

• Model Composition

MBT

USE CASE: CONTAINER TERMINAL

• Multiple components

• Components need to interact to function

• Controller needed to coordinate interaction

23

RESULTS: ASD-CONTAINER TERMINAL

24

Higher overall productivity compared to traditional development.

Main remaining problems:

- Testing interaction and complete system (not fully supported)

- Data interaction not fully supported

- Debugging of third party library (some bugs found in manual written

code, legacy code) Testing still necessary

Productivity

Method

OVERVIEW OF PROJECT STAGES

25

Use case

/SUT

Goal

Technique

Tool

Results

Stage 1

Container Terminal (CT)

Development of control

software for the CT

Model Driven Engineering +

hand written sw

ASD:Suite

Create design Models +

interface models in ASD

Generate code

Productivity (code

generation) c.t. trad. dev

-

Testing complete system,

Interaction, data, external

code

Stage 2


Verification of the

developed control software

of the CT (generated +

Hand written)

Model based Testing

Spec Explorer

Create Test Model in Spec

Explorer, generate test

suite

Test Productivity

Modeling skills,

complexity costs

Stage 3


Verification of the developed control

software of the CT (generated + Hand

written)

Model based Testing

ASDspec & Spec Explorer

Generate Test Model from existing ASD

interface Model , complete Test Model ,

generate test suite

+ Productivity (code generation, partial test

generation)

Testing complete system, Interaction, data,

external code

(benefits only in case of existing of ASD

Models)

RESULTS: ASDSPEC- CONTAINER TERMINAL

26

Spec Explorer ASD + ASDSpec

Approach Generate test suite Generate test model

Techniques MBT MBT/MDE (M2M)

Effort/complexity Medium Low

Test cases 89 93

Perceived effectiveness Medium High

Bugs Some bugs in hand written code

/HAL

All known bus found

• Low effort model creation

- Easy reuse of existing work

• Testing of complete system

- Legacy code, data interaction

• High Quality, Cost Reduction

GENERAL MDSD ARCHITECTURE

Model

Reverse

Engineering

Model

Repository

Model Code,

Doc, etc.

Model

Editor(s)

Model

Checker

Model Transformation

M2M M2T

Metamodel(s)

27

ASDSPEC ARCHITECTURE

ASDSpec

makes use of

existing MDSD

architecture

Model

Reverse

Engineering

Model

Repository

Spec

Explorer

Model

SpecExplorer

Cord

Model

Editor(s)

Model

Checker

Model Transformation

M2M M2T

Metamodel(s)

ASD

Interface

models

28

ASDSPEC: TECHNOLOGIES

QVTO ASD

XML

SE

model

Cord

script

Acceleo

Acceleo

ASD

EMF

SE

EMF

Cord

EMF

QVTO

QVTO

ASDSpec

Model transformations bridge gap between ASD

and Spec Explorer

29

TOOL STATUS

• Tool as an standalone application.

• (Pending) tool will be available as an open source Eclipse plugin for the Eclipse community.

• For now available on request

• Future steps

– Support data handling/configuration aspects in ASDSpec, instead of relying on manual additions in Spec Explorer,

– Supporting other MBT tools.

30

ASDSPEC: USING BEST OF BOTH WORLDS

• Automatic generation of Spec Explorer test model from existing ASD interface model

Benefits

• Less effort for model creation

– reuse of existing work

• Testing of complete system including

– Legacy code

– External components

– Data combination testing

– Interaction testing

• Results: High Quality, Reduced cost

Statistical Dynamic

ASDspec

31

CONCLUSIONS

• MDE and MBT technologies have matured a lot in the

latest years

• It is a matter of time….. Evolution…..

32

Using Formal Specifications to

Support Model Based Testing

ASDSpec: A Tool Combining the

Best of Two Techniques

ASD

Static Dynamic

ASDspec

Static Dynamic

ASDspec

34

Source: On the Use of Verification and

Quantitative

Analysis in Industry

Jozef Hooman

Radboud University Nijmegen

&

Embedded Systems Institute (ESI),

Eindhoven

26 October 2012

TESTING STILL NECESSARY


35

Source: MDD and its impact on

testing...

A nanotech case study

ASD case

Bryan Bakker

EuroSTAR

May 2012


36

Source: Ultrafast development of

an Ultra Fast Scanner

Model-Driven Development Day

2011-04-29

Hans Spitshuis

rachid kherrazi-testing-asd-interface-compliance-with-asd spec

Software