vdmunit testing, combinatorial testing, model quality and code generation professor peter gorm...

VDMUnit Testing, Combinatorial Testing, Model Quality and Code Generation

Professor Peter Gorm LarsenDepartment of Engineering, Aarhus University

(pgl@eng.au.dk) and

PhD student Peter Jørgensen (pvj@eng.au.dk)

2

Agenda

The VDMUnit Testing Classes

• The Combinatorial Testing Principles

• Internal Consistency

• External Consistency

• Code Generation

• Project Proposals

3

The VDMUnit Framework

4

The TestSuite Class

class TestSuite is subclass of Test

instance variables tests : seq of Test := [];

operations public Run: () ==> () Run () == (dcl ntr : TestResult := new TestResult(); Run(ntr); ntr.Show()); public Run: TestResult ==> () Run (result) == for test in tests do test.Run(result);

public AddTest: Test ==> () AddTest(test) == tests := tests ^ [test];

end TestSuite

The TestCase Class

public Run: TestResult ==> ()Run (ptr) == trap <FAILURE> with ptr.AddFailure(self) in (SetUp(); RunTest(); TearDown()); protected SetUp: () ==> ()SetUp () == is subclass responsibility;protected RunTest: () ==> ()RunTest () == is subclass responsibility;protected TearDown: () ==> ()TearDown () == is subclass responsibility

end TestCase

class TestCase is subclass of Test

instance variables name : seq of char

operations

public TestCase: seq of char ==> TestCaseTestCase(nm) == name := nm;

public GetName: () ==> seq of charGetName () == return name;

protected AssertTrue: bool ==> ()AssertTrue (pb) == if not pb then exit <FAILURE>;

protected AssertFalse: bool ==> ()AssertFalse (pb) == if pb then exit <FAILURE>;

6

The TestResult Class

class TestResult

instance variables failures : seq of TestCase := [] operations public AddFailure: TestCase ==> () AddFailure (ptst) == failures := failures ^ [ptst];

public Print: seq of char ==> () Print (pstr) == def - = new IO().echo(pstr ^ "\n") in skip; public Show: () ==> () Show () == if failures = [] then Print ("No failures detected") else for failure in failures do Print (failure.GetName() ^ " failed") end TestResult

7

Enigma Unit Tests

8

Agenda

The VDMUnit Testing Classes

The Combinatorial Testing Principles

• Internal Consistency

• External Consistency

• Code Generation

• Project Proposals

9 9

Combinatorial Testing Perspective

Regular expression

Overview of results

Detailed test case and results

10

Combinatorial Testing Overview

VDM source Parser

ASTPattern

tree

Fully expanded

tests

Shape reduced

tests

Filtered test

results

T: let x in set {1, ..., 100} in obj.op(x){1,3};

11

Test case execution

Test cases Test engine

Check

if

filt

ere

d b

y o

ther

test

case

s

Failed

Inconclusive

Passed

Filtered

Test results

Filter out test cases with a prefix resulting in a test failure

12

operationspublic insert : int ==> ()insert(val)== skippre val > 1;

tracesT1 : let x in set {1,2} in insert(x);

13

class Avaluesobj : A = new A();

operationspublic op : nat ==> natop (x) == return x;

tracesT2: let x,y in set {1, ..., 10}in (obj.op(x);obj.op(y));

end A

14

Regular Expressions in Traces

• Let definitions (local naming)• Let be such that definitions (all selection)• Repeat traces (fixed or variable number of times)• Concurrency traces• Application expression (calling operations)

15

Agenda

The VDMUnit Testing Classes

The Combinatorial Testing Principles

Internal Consistency

• External Consistency

• Code Generation

• Project Proposals

16

Introduction

• What is the value of the models you have produced?

• How do we assess the quality of a model?• Internal consistency:

• Does the model describe something?• Syntax, type checking and proof obligations• No potential run-time errors

• External consistency: • Does the model describe the right thing?• Validation with domain expert• Does the model have desirable properties?

17

POP3: Protection of Partial Operatorsclass POP3Server...instance variables maildrop : MailDrop; ...

types

public MailDrop = map POP3Types`UserName to MailBox;public MailBox :: msgs : seq of POP3Message locked : bool;

operations

GetUserMessages: POP3Types`UserName ==> seq of POP3MessageGetUserMessages(user) == return GetUserMail(user).msgspre UserKnown(user);

end POP3Server

18

Booking of Flights: Invariant Preservationclass Trip

typesFlight :: departure : seq of char destination : seq of char

instance variables journey: seq of Flight;

inv forall i in set {1,...,len journey -1} & journey(i).destination = journey(i+1).departure

operations

AddFlight: Flight ==> ()AddFlight(f) == journey := journey ^ [f]pre journey(len journey).destination = f.departure

end Trip

journey <> [] =>

19

Robot Routes: Satisfiability 1

class Routeinstance variables

points: set of Point;

inv forall p1, p2 in set points & p1.GetCoord() = p2.GetCoord() => p1 = p2 and forall p in set points & p.GetIndex() <> card points => GetNext(p).GetCoord() in set {n.GetCoord() | n in set p.Neighbour()}…end Route

A pre/post condition specification is satisfiable, if, for all states and inputs satisfying the precondition, there exists some output and “after” state satisfying the postcondition and any relevant invariants.

20

Robot Routes: Satisfiability 2

class Routefunctions

staticpublic AvoidanceRoutes( obstacles: set of (nat * nat), currentPosition: Point, nextWaypoint: Point) routes: set of Routepost forall r in set routes & r.GetFirst().GetCoord() = currentPosition.GetCoord() and r.GetLast().GetCoord() = nextWaypoint.GetCoord() and r.GetCoords() inter obstacles = {};

end Route

21

Robot Routes: Satisfiability 3

• For implicit definitions there must exist at least one potential result for each input satisfying the pre-condition

Proof Obligation (or integrity constraint):

forall obstacles: set of (nat * nat), currentPosition: Point, nextWaypoint: Point & exists routes: set of Route & post-AvoidanceRoutes(obstances,currentPosition, nextWaypoint,routes)

Can in principle be proved formally

22

Agenda

The VDMUnit Testing Classes

The Combinatorial Testing Principles

Internal Consistency

External Consistency

• Code Generation

• Project Proposals

23

Dialogue with Domain Experts

• Typically domain experts know little about IT• Understanding their intended usage may be a

challenge• Creating a model will create further questions to

experts• Model should seldom be shown directly• Scenarios to be used for test purposes can

typically be discussed• A Java-based API can be used to ”demonstrate”

ideas to domain experts/end users

24

The External Interfaces of Overture

• External aid is needed to support communication.

• Overture supports two different types of external interfacing• External Call Interface

• Only calls from the VDM model to an external interface

• Remote Control Interface• Allows for external calls into a

VDM Model

25

External Call Interface• Enables call to native Java code from inside a model• A type of delegate scheme is created between the

model and an external jar• In the model an operation is created which contains

an "is not yet specified" statement

public receivedMessage: int ==> () receivedMessage(vecID) == is not yet specified;

• When a non-specified statement is encountered, the interpreter will considered it an delegate, an attempt to look it up in Java CLASSPATH.

26

External Call Interface• The Java class must have the same name as the module

or class. • Package separators are denoted with underscores in the

model.VDM:class gui_GraphicsJava:package gui;class Graphics

• If a Java class and method is found, a delegate object is created an bound to the delegate function.

• MATH, IO and VDMUtil standard library functions in Overture are implemented using the same delegate scheme

27

External Call Interface Example : VDM

class gui_Graphics public receivedMessage : int ==> () receivedMessage(vecID) == is not yet specified;end gui_Graphics

28

External Call Interface Example : Javapackage gui;

import org.overture.interpreter.runtime.ValueException;import org.overture.interpreter.values.Value;import org.overture.interpreter.values.VoidValue;

public class Graphics implements Serializable { public Value receivedMessage(Value vecID) throws ValueException { model.receivedMessage(vecID.intValue(null)); return new VoidValue(); }}

29

External Call Interface Example : Java IDE• In order to use VDM Types the Java program must

have the Overture java library in its build path.• ast.jar, parser.jar and interpreter.jar must be

available on the built path.• These jar files are provided with Overture

• <Overturedir>\Plugins\• Once the Java program is finished, it must be exported to a

Jar file and placed in the lib directory in the Overture/VDM projects directory, in order for Overture to find it.

30

External Call Interface Example

31

Remote Control Interface• Allows for a VDM model to be controlled by an

external Java programs• Essentially a console like access to the Overture

interpreter which can be bootstrapped through the Overture Debugger

• A RemoteControl interface is defined, which must be implemented by the external programs.public interface RemoteControl{

public void run(RemoteInterpreter interpreter) throws Exception;

}

32

RemoteInterpreter• RemoteInterpreter is passed from Overture to the

external programs • It has four central methods

• Void Init()• Re-initializes the interpreter

• Void Create(String, String)• Takes a variable name as the first argument

and expression as the second. • String Execute(String)

• Takes expression as string argument, returns a string value.

• Value ExecuteValue(String)• Takes expression as string argument, returns a

generic VDM Value represented in Java

33

Example model to use

class A

operations

public op: int ==> intop(n) ==

return n + 1pre n > 0

end A

34

Use of the RemoteInterpreter

interpreter.init()

interpreter.create(("a", "new A()"); //create a := new A();

Value result = interpreter.valueExecute("a.op(5)");

System.out.println(result.intValue(null)); //prints 6

35

Use of the Remote Control Interface• Using the Remote Control Interface to create an

interactive GUI for a model

• Connecting a GUI to the KLV /CSLaM model• Create a GUI• Implement RemoteControl interface• Bind GUI with the RemoteInterpreter through the

RemoteControl.

36

Remote Control Interface Example (RemoteControl)import org.overture.interpreter.debug.RemoteControl;import org.overture.interpreter.debug.RemoteInterpreter

public class KlvRemote implements RemoteControl{@Overridepublic void run(RemoteInterpreter intrprtr){

Thread remoteThread = new Thread(new Runnable(){ public void run(){ KLVgui gui = new KLVgui(); KLVgui.vdmklv = new KlvOvertureComm(intrprtr, gui); gui.initialising(); }); remoteThread.setDaemon(true); remoteThread.start(); intrprtr.processRemoteCalls();}

37

Remote Control Interface Example (KlvOvertureComm)public int getMaxSpeed() { try{

Value g = execute("klv.getMaxSpeed()");return new Long(g.intValue(null)).intValue();

} catch (Exception err) {printError("API error: " + err.getMessage());}

return 0;}

private Value execute(String arguments) throws Exception{System.out.println("Calling Overture with: " + cmd);Value result = interpreter.valueExecute(cmd);

return result; }

38

Setting up the Remote Control Interface• External Java class must be configured in the Overture

Debug Configurations

Validating KLV using the API

40

Agenda

The VDMUnit Testing Classes

The Combinatorial Testing Principles

Internal Consistency

External Consistency

Code Generation

• Project Proposals

41

Overture Code Generation• A VDM++ to Java Code Generator • Available in Overture releases 2.0.6 onwards• Early work: VDM++ to C++ Code Generator

42

Code generation challenges (Java)

VDM:public op : () ==> natop () == ( dcl v1 : Vector2D := mk_Vector2D(1,2); dcl v2 : Vector2D := v1; v1.x := 2; return v2.x;)

Java:public Number op() { Vector2D v1 = new Vector2D(1L, 2L); Vector2D v2 = v1; v1.x = 2L; return v2.x;}

• Code generating VDM++ models to Java• Preserving the semantics in the generated code• Code generation across paradigms

• How do we generate: {x | x in set S & pred(x)}• Or Lambdas, e.g. (lambda x : int & (lambda y : int & x + y))

Java (correct):public Number op() { Vector2D v1 = new Vector2D(1L, 2L); Vector2D v2 = v1.clone(); v1.x = 2L; return v2.x;}

43

Example: Set comprehensions

public f : () -> set of natf () ==let a = {x | x in set S & pred(x)}in g(a,a);

public op : () ==> set of natop () ==( dcl setCompResult : set of nat := {}; for all x in set S do if pred(x) then setCompResult := setCompResult union {x}; (dcl a : set of nat := setCompResult; return g(a,a)));

44

Code Generation Platform

45

Agenda

The VDMUnit Testing Classes

The Combinatorial Testing Principles

Internal Consistency

External Consistency

Code Generation

Project Proposals

46

Project Proposals• R&D projects (5 ECTS) or MSc thesis (30 ECTS)• Validating a code generator for embedded systems

• Code generate for an embedded platform• NXT, Raspberry Pi etc.• Identify and address issues

• Validating the code generation platform architecture• Adding (partial) support for a new language• Identify changes/extensions to the IR• Feedback for the platform architecture

• Code Generating VDM-SL models• Converting a VDM-SL model into a VDM++ model• Without changing the code generator• Implementation of the conversion rules

47

Project Proposals• Code Generating Union Types

• How can union types be represented in (say) Java• Comparing different approaches• Implementing support for union types

• Code Generating VDM-RT• In VDM-RT objects can be deployed on different CPUs• CPUs communicate via buses• Objects message exchange results in bus communication

• Code generation of VDM concurrency concepts• Other projects:

http://wiki.overturetool.org/index.php/Overture_projects#Additional_Potential_Projects_for_.22Technical_IT.22_students_at_Aarhus_University

48

Summary• What have I presented today?

• Introduced VDMUnit for testing• Introduced combinatorial testing principles• Assessing model quality• Internal consistency• External consistency• Code Generation• Project Proposals

• What do you need to do now?• Read chapter 13• Assess your own mini-project VDM model’s consistency• Carry out either the testing exercise or the exercise with a

graphical front-end using java

49

Quote of the day

Bertrand Meyer

Formal specifications may become for software engineers what, say, differential equations are

for engineers of other fields