lsr tobias and some ase challenges y. ledru p. bontron, o. maury, l. du bousquet, m.l. potet, c....

LSRTOBIAS

and some ASE challenges

Y. Ledru

P. Bontron, O. Maury,

L. du Bousquet, M.L. Potet,

C. Oriat , S. Beghdadi, H. Bouldjedri

LSR/IMAG – Grenoble

ASE Workshop at Irvine, June 2002

Overview• Two challenges of ASE

• TOBIAS

• How TOBIAS is confronted to these challenges

OverviewTwo challenges of ASE

• TOBIAS

Signature of ASE tools

Formal or structured document

ASEtool

Source code, tests,formal specifications,

UML diagrams,XML documents

1st challenge: process documents

• Complexity of input/output documents• Algorithmic complexity of processing:

– Combinatorial explosion– Undecidable problems

ASEtool

Challenge 1 : Process

documents

Domain specific

2nd challenge: produce documents

• Software engineers only like to write code! [XP]• Millions of amateur software engineers…• … how can we learn them category theory?

ASEtool

Captureinformation

documents

Challenge 2 : Produce

documents

Elements of solutions

• Don’t underestimate software engineers: provide useful/efficient/scalable tools!• Provide easier formalisms (graphical formalisms, GUI tools,…)• Integrate your tools in the process of the company!

ASEtool

Captureinformation

documents

Challenge 2 : Produce

documents

costs vs b

enefits

TOBIAS

• Generates a large number of tests…• … from a generative pattern (test schema)• Should be used in combination with tools that

provide the test oracle (VDMTools, TGV).

Test schemas

Test data/Test cases/

Test purposesTOBIAS

Captureinformation

Class signatures

A simple example

• An integer set classwith two operations

• « manual » tests:– Add(0)– Add(0),Add(1),Add(2)– Add(0),Add(0)– Add(0),Remove(0)– Add(0),Add(1),Remove(0)– Add(0),Remove(1)

Add(v:int)Remove(v:int)

IntegerSet

Add(0)Add(1)Add(2)Add(3)

• Add(x)^1..3;Remove(y)^0..2• Where • x : {0,1,2,3}• y : {0,1,2}

Add(0),Add(0) Add(2),Add(0)Add(0),Add(1) Add(2),Add(1)Add(0),Add(2) Add(2),Add(2)Add(0),Add(3) Add(2),Add(3)

A TOBIAS test schema

…Add(0),Add(1),Add(2)…Add(2),Add(1),Add(0)… Add(0),Remove(0)

Add(0),Remove(1)…

…Add(0),Add(1),Remove(0)…

• Add(x1)• Add(x1),Add(x2)• Add(x1),Add(x2),Add(x3)• Add(x1),Remove(y1)• Add(x1),Add(x2),Remove(y1)• Add(x1),Add(x2),Add(x3), Remove(y1)• Add(x1),Remove(y1),Remove(y2)• Add(x1),Add(x2),Remove(y1),Remove(y2)• Add(x1),Add(x2),Add(x3), Remove(y1),Remove(y2)

(4+4*4+4*4*4)*(1+3+3*3) = 1092 sequences

Results of TOBIAS• TOBIAS generates the manual tests

+ lots of similar tests

for the same design cost

• A recent experiment:– 45 manual tests– 4320 TOBIAS tests– same test design effort– 2 new errors detectedTOBIAS helps detecting more errors!

TOBIAS

How TOBIAS is confronted to these challenges

1st challenge complexity• Intrinsic to the tool:

– generates a large number of tests– because IUTs are complex

• Still too many tests are useless:– Require untractable resources– May not detect more errors

Keep the number of tests small enough:• Avoid redundant test cases

Extend the test schemas (constraints, hypotheses)

• Remove non-conform test cases Verify vs specification

Test schemas

TOBIAS and complexity

Test purposesTOBIAS

Captureinformation

Class signaturesSolutions to the

first challenge…

… make the second one more complex!

ExtendedTest

schemas

Specification(UML diagrams,

Pre/post cond., …)

Test schemas are simple and short !

TOBIAS provides a GUI for test schemas

Specificationscan be used in

conjunction with generated tests(oracle, TGV,…)

Specificationsare not neededto generate the first set of tests!

Class signaturescan be extracted

from existing documents

Capturing knowledge for TOBIAS

Test schemas

Test purposesTOBIAS

Captureinformation

Class signatures

Specification(UML diagrams,

Pre/post cond., …)

UML class diag.Or

Java Programextract

Conclusion• TOBIAS is confronted to both ASE

challenges– complexity– information acquisition

• Complexity addressed by additional knowledge

• Information acquired progressively:– Provide first benefits at low cost– Motivate efforts for specification– Multiple use of specifications (oracle, TGV)– Fits into a classical process

THANKS !

Redundant test cases• The implementation of the integer set

may be independent from the actual values of integers

• Lots of tests generated by TOBIAS can be considered redundant

• This corresponds to the expression of test hypotheses.

Add(0)Add(1)Add(2)Add(3)

…Add(0),Add(1),Add(2)…Add(2),Add(1),Add(0)…

LSR Add(x)^1..3;Remove(y)^0..2Where x : {0,1,2,3}y : {0,1,2}Forall x_i,x_j. i<j => x_i<=x_j

…Add(0),Add(1),Add(2)…Add(2),Add(1),Add(0)…

Overview

• Context: the COTE project (COmponent TEsting)

• Why TOBIAS ?

• TOBIAS: description of the tool– Test purpose generation– Test case generation with VDM– Smarter generation

Overview

• Why TOBIAS ?

The COTE project

• A RNTL Project

• Partners – SOFTEAM (tool objecteering) leader– IRISA (tools UMLAUT/TGV)

– GEMPLUS (smart cards/software components)

– FT R&D (OO technologies/components)

– LSR-IMAG (Software engineering)

The context of COTE

• Conformance testing of software components– The component is tested against its UML

specification

– Black box testing: the internal behaviour of the product under test is hidden

Aim of the COTE project

corba ejb .net

UMLTest casesAutomatic

generationof executable tests

• UML sequence diagrams provide an abstraction for tests cases

• provide tools for automatic executable test generation from UML test cases

• target three component technologies

Aim of the COTE project

Testsynthesis

• Assist the user in the specification of tests

• UML specification of tests

corba ejb .net

UMLTest casesAutomatic

generationof executable tests

test purpose

Test case / Test purposeAn example :

A coffee vending machine

!sugar

?coffee+sugar ?coffee+sugar

?coin?coin

!coffee

!sugar

A test purposea sequence of transitions to reach

?coffee

?tea+sugar

?coffee+sugar

!coffee !coffee

!sugar

?i : input of the specification

!o : output of the specification

A test casea complete path in the behavioural specification

General structure of COTE

UMLTest purposes

Behaviour computationUMLAUT/TGV

UMLTest cases

corba ejb .net

Specification

Testsynthesis

Automatictests

generation

Overview

• Why TOBIAS ?

Why TOBIAS ?Behavioural testing requires the specification of

many tests.– Test campaign => (potentially) thousands of test

cases– UMLAUT/TGV:

one test purpose and one execution = one test case

– New values => creation of new test purposes

Hugues MARTIN’s doctoral thesis (GEMPLUS): « Une méthodologie de génération automatique

de suites de tests pour applets java-card. » A methodology for automatic generation of

test suites for java-card applets.

Generation of test purposes

• Idea : notion of high level test purpose

(test pattern)

=> abstraction of test purposes on objects, methods or values…

• A test pattern => a set of test purposes => a set of test cases

Significantly reduces the repetitive tasks of test definitions

TOBIAS

Specification

UMLTest purposes

Behaviour computationUMLAUT/TGV

Test synthesis

UMLTest cases

TOBIAS

Test pattern

Overview

• Why TOBIAS ?

Principles of TOBIAS• TOBIAS provides:

– help for test purposes design– test pattern descriptions

• Idea:– instantiation of test patterns according to

different criteria (objects, parameters …)

TOBIAS inputs

test patterns

UML specification

class diagram

deployment diagram

Test purposein UML

TOBIAS

A little example (1)•UML class diagram

IAdministrator IUser HCI

VendingMachine

Additionnal Drink

Cancel

Button

NumButton : integer

PushButton()

insertCoin(Coin p)

GiveChange()

A little example (2)• UML deployment diagram (one for each

test campaign)

TestConfiguration:

DrinkMachine:VendingMachine

Testor:Actor

Front:HCI

Coffee:Drink

Tea:Drink

Cappucino:Drink

Chocolate:Drink

Reset:Cancel

Some test purposes

For:- 4 different drinks- 1 cancel touch (Reset.GiveChange)- 5 different coins = {0.1, 0.2, 0.5, 1, 2}

The number of different test purposes is around:(5+5²) x (4 + 1) = 150 test purposes

Coffee.ClickButton()Chocolate.ClickButton()Tea.ClickButton()Cappucino.ClickButton()Reset.GiveChange()

insertCoin(0.2€);insertCoin(0.1€);…

1. insertCoin(2€);2. insertCoin(0.5€);3. insertCoin(1€);4. insertCoin(0.1€);5. insertCoin(0.2€);

5 test purposes :

Principle of test patterns• Provide the generation of test purposes

according to different criteria:– the method parameters– the objects– the groups– the number of group or method calls

An example

GroupAction = { Drink.ClickButton(), Cancel.GiveChange()}

Using the following test pattern:

insertCoin(x)^1..2 ; GroupAction

TOBIAS generates the 150 test purposes previously showed

Advantages of using TOBIAS

• TOBIAS is useful to generate many test purposes

generating many test cases using UMLAUT/TGV

• Generating test purposes for new values is easier

• TOBIAS can be used to generate test suites for VDM in combination with VDMTools

Using TOBIAS with VDM

• VDMtools provide an environment to specify an application, to execute test suites and to evaluate the test results

• The output of TOBIAS is directly used as a test suite.

Using TOBIAS for the automatic generation of VDM test cases - VDM Workshop 2002 at Floc02

TOBIAS

VDMTools

UML specification

Test patter

VDM test suites

Results: • fault detection• code coverage

VDM specification

• We did the following experiment:– specify an application: managing groups of

students– manually specify a test suite

(about 15 test cases)– generate a test suite using TOBIAS

(about 4300 test cases)

Using TOBIAS with VDM• We obtained the following results:

– the generated and the manually written test suite have the same code coverage

– The development effort is similar

– the generated test suite detect more errors• one unknown error• one known error (from reading the specification)• new activations of a known error

– the generated test suite takes significantly more time to be executed:

• TOBIAS generates non conforming test cases• TOBIAS generates redundant test cases

Smarter generation• Preventing non conforming test

purposes / test cases– using more elements from the

specification• relations between classes• pre/post conditions• state diagrams

• Reducing the test suite size– eliminating redundant test purposes / test

cases– defining constraints on the schemas

Using more elementsTOBIAS extract classes from the UML class diagram.

Class A

Class B Class D

Class C

A!C C!A B!C C!B A!B B!A

A!D D!A B!D D!B C!D D!C

A!A B!B C!C D!D

the class A calls the methods of the class C

Using more elementsRelations between classes

A!C C!A B!C xC!B xA!B xB!A x

Class A

Class B Class D

Class C0..1

A!D xD!A B!D D!B C!D xD!C x

A!A xB!B xC!C xD!D x

Using more elements

Deployment diagram that gives a fixed state of the application

b:B d:D

A!C a!cC!A c!aB!C xC!B xA!B xB!A x

A!D xD!A d!a xB!D b!dD!B d!bC!D xD!C x

Deployment diagram

Using more elements

Deployment diagram that gives the initial state of the application

b:B d:D

A!C a!cC!A c!aB!C xC!B xA!B xB!A x

A!D xD!A d!aB!D b!dD!B d!bC!D xD!C x

Adding constraints

• The user can associate a constraint to each test pattern

• Constraints deal with parameters

• Constraints are used to filter test purposes / test cases

Adding constraints

)1_2_)6.02_1_(()2_var( xxxxx

A test pattern:

insertCoin(x)^1..2 ; GroupAction

GroupAction = { Drink.ClickButton(), Cancel.GiveChange()}

A constraint:

Number of selected test purposes:

(5+9)x(4+1) = 70 test purposes

- insertCoin(x_1) ; GroupAction- insertCoin(x_1) ; insertCoin(x_2) ; GroupAction

Conclusion• TOBIAS provides a new abstract level for test design: the

test patterns

• A test pattern => a set of similar test cases

• To avoid combinatorial explosion, the user can associate

constraints to the schemas

• TOBIAS can be used to generate test sequences

• TOBIAS will be used and validated on an industrial test case

in the COTE project

Prospects

• Refining the test generation

– Deleting redundant

– Using pre/post conditions on operation to eliminate

non-conforming sequences

• Helping the decision to stop test campaign:

– Specification coverage by test patterns using more elements from the

specification

lsr tobias and some ase challenges y. ledru p. bontron, o. maury, l. du bousquet, m.l. potet, c....

Documents

michael hughes - lsr

lsr cyclon repair manual

clec operations: suddenlink lsr gui...lsr is designed to...

innovations in lsr molding - new materials &...

bd lsr fortessa x20 how to use - igc - facilities...

lsr f. oÖ ddb 09.09.2004 1 lsr f. oÖ ddb 09. september...

update on lsr activities - cochrane · update on lsr...

uc weedric - c: rice field daywric.ucdavis.edu/pdfs/2015...

lsr test purposes: adapting the notion of specification to...

lsr 1 jartege : a tool for random generation of unit tests...

lsr - racing history project

1800 lsr manual - server login manuals/2001/1800 lsr...

1 objectweb/osmose deployment framework didier donsez,...

lsr asx announcement · 336,300,364 (lsr) options on issue:...

lsr newsletter july 2021

seebeck coefficient lsr electric resistance zt …€¦ ·...

bd lsr ii user’s guide - atlanta pediatric research ·...

signal processing: image...

pernoud lsr overmolding process

lsr vpresntation