lsr tobias and some ase challenges y. ledru p. bontron, o. maury, l. du bousquet, m.l. potet, c....

LSRTOBIAS

and some ASE challenges

Y. Ledru

P. Bontron, O. Maury,

L. du Bousquet, M.L. Potet,

C. Oriat , S. Beghdadi, H. Bouldjedri

LSR/IMAG – Grenoble

ASE Workshop at Irvine, June 2002

LSR

Overview• Two challenges of ASE

• TOBIAS

• How TOBIAS is confronted to these challenges

LSR

OverviewTwo challenges of ASE

• TOBIAS


LSR

Signature of ASE tools

Formal or structured document


ASEtool

Source code, tests,formal specifications,

UML diagrams,XML documents

LSR

1st challenge: process documents

• Complexity of input/output documents• Algorithmic complexity of processing:

– Combinatorial explosion– Undecidable problems



ASEtool

Challenge 1 : Process

documents

Domain specific

tools

LSR

2nd challenge: produce documents

• Software engineers only like to write code! [XP]• Millions of amateur software engineers…• … how can we learn them category theory?



ASEtool

Captureinformation


documents

Challenge 2 : Produce

documents

LSR

Elements of solutions

• Don’t underestimate software engineers: provide useful/efficient/scalable tools!• Provide easier formalisms (graphical formalisms, GUI tools,…)• Integrate your tools in the process of the company!



ASEtool

Captureinformation


documents

Challenge 2 : Produce

documents

costs vs b

enefits

LSR


TOBIAS


LSR

TOBIAS

• Generates a large number of tests…• … from a generative pattern (test schema)• Should be used in combination with tools that

provide the test oracle (VDMTools, TGV).

Test schemas

Test data/Test cases/

Test purposesTOBIAS

Captureinformation

Class signatures

LSR

A simple example

• An integer set classwith two operations

• « manual » tests:– Add(0)– Add(0),Add(1),Add(2)– Add(0),Add(0)– Add(0),Remove(0)– Add(0),Add(1),Remove(0)– Add(0),Remove(1)

Add(v:int)Remove(v:int)

IntegerSet

LSR

Add(0)Add(1)Add(2)Add(3)

• Add(x)^1..3;Remove(y)^0..2• Where • x : {0,1,2,3}• y : {0,1,2}

Add(0),Add(0) Add(2),Add(0)Add(0),Add(1) Add(2),Add(1)Add(0),Add(2) Add(2),Add(2)Add(0),Add(3) Add(2),Add(3)


A TOBIAS test schema

…Add(0),Add(1),Add(2)…Add(2),Add(1),Add(0)… Add(0),Remove(0)

Add(0),Remove(1)…

…Add(0),Add(1),Remove(0)…

• Add(x1)• Add(x1),Add(x2)• Add(x1),Add(x2),Add(x3)• Add(x1),Remove(y1)• Add(x1),Add(x2),Remove(y1)• Add(x1),Add(x2),Add(x3), Remove(y1)• Add(x1),Remove(y1),Remove(y2)• Add(x1),Add(x2),Remove(y1),Remove(y2)• Add(x1),Add(x2),Add(x3), Remove(y1),Remove(y2)

(4+4*4+4*4*4)*(1+3+3*3) = 1092 sequences

LSR

Results of TOBIAS• TOBIAS generates the manual tests

+ lots of similar tests

for the same design cost

• A recent experiment:– 45 manual tests– 4320 TOBIAS tests– same test design effort– 2 new errors detectedTOBIAS helps detecting more errors!

LSR


TOBIAS

How TOBIAS is confronted to these challenges

LSR

1st challenge complexity• Intrinsic to the tool:

– generates a large number of tests– because IUTs are complex

• Still too many tests are useless:– Require untractable resources– May not detect more errors

Keep the number of tests small enough:• Avoid redundant test cases

Extend the test schemas (constraints, hypotheses)

• Remove non-conform test cases Verify vs specification

LSR

Test schemas

TOBIAS and complexity


Test purposesTOBIAS

Captureinformation

Class signaturesSolutions to the

first challenge…

… make the second one more complex!

ExtendedTest

schemas

Specification(UML diagrams,

Pre/post cond., …)

LSR

Test schemas are simple and short !

TOBIAS provides a GUI for test schemas

Specificationscan be used in

conjunction with generated tests(oracle, TGV,…)

Specificationsare not neededto generate the first set of tests!

Class signaturescan be extracted

from existing documents

Capturing knowledge for TOBIAS

Test schemas


Test purposesTOBIAS

Captureinformation

Class signatures

Specification(UML diagrams,

Pre/post cond., …)

UML class diag.Or

Java Programextract

LSR

Conclusion• TOBIAS is confronted to both ASE

challenges– complexity– information acquisition

• Complexity addressed by additional knowledge

• Information acquired progressively:– Provide first benefits at low cost– Motivate efforts for specification– Multiple use of specifications (oracle, TGV)– Fits into a classical process

LSR

THANKS !

LSR

Redundant test cases• The implementation of the integer set

may be independent from the actual values of integers

• Lots of tests generated by TOBIAS can be considered redundant

• This corresponds to the expression of test hypotheses.

Add(0)Add(1)Add(2)Add(3)

…Add(0),Add(1),Add(2)…Add(2),Add(1),Add(0)…



LSR Add(x)^1..3;Remove(y)^0..2Where x : {0,1,2,3}y : {0,1,2}Forall x_i,x_j. i<j => x_i<=x_j



…Add(0),Add(1),Add(2)…Add(2),Add(1),Add(0)…

LSR

Overview

• Context: the COTE project (COmponent TEsting)

• Why TOBIAS ?

• TOBIAS: description of the tool– Test purpose generation– Test case generation with VDM– Smarter generation

LSR

The COTE project

• A RNTL Project

• Partners – SOFTEAM (tool objecteering) leader– IRISA (tools UMLAUT/TGV)

– GEMPLUS (smart cards/software components)

– FT R&D (OO technologies/components)

– LSR-IMAG (Software engineering)

LSR

The context of COTE

• Conformance testing of software components– The component is tested against its UML

specification

– Black box testing: the internal behaviour of the product under test is hidden

LSR

Aim of the COTE project

corba ejb .net

UMLTest casesAutomatic

generationof executable tests

• UML sequence diagrams provide an abstraction for tests cases

• provide tools for automatic executable test generation from UML test cases

• target three component technologies

LSR

Aim of the COTE project

Testsynthesis

• Assist the user in the specification of tests

• UML specification of tests

corba ejb .net

UMLTest casesAutomatic

generationof executable tests

test purpose

LSR

Test case / Test purposeAn example :

A coffee vending machine

!sugar

?coffee+sugar ?coffee+sugar

?coin?coin

!coffee

!sugar

A test purposea sequence of transitions to reach

?tea

?coffee

?tea+sugar

?coffee+sugar

?coin

?coin

!coffee !coffee

!tea

!sugar

!tea

!sugar

?i : input of the specification

!o : output of the specification

A test casea complete path in the behavioural specification

LSR

General structure of COTE

UMLTest purposes

Behaviour computationUMLAUT/TGV

UMLTest cases

corba ejb .net

UML

Specification

Testsynthesis

Automatictests

generation

LSR

Overview


• Why TOBIAS ?


LSR

Why TOBIAS ?Behavioural testing requires the specification of

many tests.– Test campaign => (potentially) thousands of test

cases– UMLAUT/TGV:

one test purpose and one execution = one test case

– New values => creation of new test purposes

Hugues MARTIN’s doctoral thesis (GEMPLUS): « Une méthodologie de génération automatique

de suites de tests pour applets java-card. » A methodology for automatic generation of

test suites for java-card applets.

LSR

Generation of test purposes

• Idea : notion of high level test purpose

(test pattern)

=> abstraction of test purposes on objects, methods or values…

• A test pattern => a set of test purposes => a set of test cases

Significantly reduces the repetitive tasks of test definitions

LSR

TOBIAS

UML

Specification

UMLTest purposes

Behaviour computationUMLAUT/TGV

Test synthesis

UMLTest cases

TOBIAS

Test pattern

LSR

Overview


• Why TOBIAS ?


LSR

Principles of TOBIAS• TOBIAS provides:

– help for test purposes design– test pattern descriptions

• Idea:– instantiation of test patterns according to

different criteria (objects, parameters …)

LSR

TOBIAS inputs

test patterns

UML specification

class diagram

deployment diagram

Test purposein UML

TOBIAS

LSR

A little example (1)•UML class diagram

Actor

IAdministrator IUser HCI

*1

1

1

1

VendingMachine

Additionnal Drink

Cancel

Button

NumButton : integer

PushButton()

insertCoin(Coin p)

GiveChange()

1..5

1

0..1

LSR

A little example (2)• UML deployment diagram (one for each

test campaign)

TestConfiguration:

DrinkMachine:VendingMachine

Testor:Actor

Front:HCI

Coffee:Drink

Tea:Drink

Cappucino:Drink

Chocolate:Drink

Reset:Cancel

LSR

Some test purposes

For:- 4 different drinks- 1 cancel touch (Reset.GiveChange)- 5 different coins = {0.1, 0.2, 0.5, 1, 2}

The number of different test purposes is around:(5+5²) x (4 + 1) = 150 test purposes

Coffee.ClickButton()Chocolate.ClickButton()Tea.ClickButton()Cappucino.ClickButton()Reset.GiveChange()

insertCoin(0.2€);insertCoin(0.1€);…

1. insertCoin(2€);2. insertCoin(0.5€);3. insertCoin(1€);4. insertCoin(0.1€);5. insertCoin(0.2€);

5 test purposes :

LSR




Principle of test patterns• Provide the generation of test purposes

according to different criteria:– the method parameters– the objects– the groups– the number of group or method calls

LSR






LSR

An example

GroupAction = { Drink.ClickButton(), Cancel.GiveChange()}

Using the following test pattern:

insertCoin(x)^1..2 ; GroupAction

TOBIAS generates the 150 test purposes previously showed

LSR

Advantages of using TOBIAS

• TOBIAS is useful to generate many test purposes

generating many test cases using UMLAUT/TGV

• Generating test purposes for new values is easier

• TOBIAS can be used to generate test suites for VDM in combination with VDMTools

LSR

Using TOBIAS with VDM

• VDMtools provide an environment to specify an application, to execute test suites and to evaluate the test results

• The output of TOBIAS is directly used as a test suite.

Using TOBIAS for the automatic generation of VDM test cases - VDM Workshop 2002 at Floc02

LSR


TOBIAS

VDMTools

UML specification

Test patter

n

VDM test suites

Results: • fault detection• code coverage

VDM specification

LSR


• We did the following experiment:– specify an application: managing groups of

students– manually specify a test suite

(about 15 test cases)– generate a test suite using TOBIAS

(about 4300 test cases)

LSR

Using TOBIAS with VDM• We obtained the following results:

– the generated and the manually written test suite have the same code coverage

– The development effort is similar

– the generated test suite detect more errors• one unknown error• one known error (from reading the specification)• new activations of a known error

– the generated test suite takes significantly more time to be executed:

• TOBIAS generates non conforming test cases• TOBIAS generates redundant test cases

LSR

Smarter generation• Preventing non conforming test

purposes / test cases– using more elements from the

specification• relations between classes• pre/post conditions• state diagrams

• Reducing the test suite size– eliminating redundant test purposes / test

cases– defining constraints on the schemas

LSR

Using more elementsTOBIAS extract classes from the UML class diagram.

Class A

Class B Class D

Class C

A!C C!A B!C C!B A!B B!A

A!D D!A B!D D!B C!D D!C

A!A B!B C!C D!D

the class A calls the methods of the class C

LSR

Using more elementsRelations between classes

A!C C!A B!C xC!B xA!B xB!A x

Class A

Class B Class D

Class C0..1

0..1

1

1

0..1

1

A!D xD!A B!D D!B C!D xD!C x

A!A xB!B xC!C xD!D x

LSR

Using more elements

Deployment diagram that gives a fixed state of the application

a:A

b:B d:D

c:C

A!C a!cC!A c!aB!C xC!B xA!B xB!A x

A!D xD!A d!a xB!D b!dD!B d!bC!D xD!C x


Deployment diagram

LSR

Using more elements

Deployment diagram that gives the initial state of the application

a:A

b:B d:D

c:C

A!C a!cC!A c!aB!C xC!B xA!B xB!A x

A!D xD!A d!aB!D b!dD!B d!bC!D xD!C x


LSR

Adding constraints

• The user can associate a constraint to each test pattern

• Constraints deal with parameters

• Constraints are used to filter test purposes / test cases

LSR

Adding constraints

)1_2_)6.02_1_(()2_var( xxxxx

A test pattern:

insertCoin(x)^1..2 ; GroupAction

GroupAction = { Drink.ClickButton(), Cancel.GiveChange()}

A constraint:

Number of selected test purposes:

(5+9)x(4+1) = 70 test purposes

- insertCoin(x_1) ; GroupAction- insertCoin(x_1) ; insertCoin(x_2) ; GroupAction

LSR

Conclusion• TOBIAS provides a new abstract level for test design: the

test patterns

• A test pattern => a set of similar test cases

• To avoid combinatorial explosion, the user can associate

constraints to the schemas

• TOBIAS can be used to generate test sequences

• TOBIAS will be used and validated on an industrial test case

in the COTE project

LSR

Prospects

• Refining the test generation

– Deleting redundant

– Using pre/post conditions on operation to eliminate

non-conforming sequences

• Helping the decision to stop test campaign:

– Specification coverage by test patterns using more elements from the

specification

lsr tobias and some ase challenges y. ledru p. bontron, o. maury, l. du bousquet, m.l. potet, c....

Documents