chaining test cases for reactive system testing test cases for reactive system testing peter...

Chaining Test Cases for Reactive System Testing

Peter Schrammel, Tom Melham and Daniel [email protected]

The 25th IFIP International Conference onTesting Software and Systems (ICTSS’13)

Nov 13-15, 2013, Istanbul, Turkey

Test Chains

Context:

Safety critical embedded software

Often modelled as synchronous reactive system

Safety standards: tool support for systematic testing desirable

Problem:

Often lengthy input sequences required to drive the system toa test goal

Reset after each test case: serious problem in on-target testing

Goal:

Find a test case chain: a single test case that covers a set oftest goals and minimises overall test execution time

Model-Based Testing

requirements design model implementationcheck check

Model-Based Testing

requirements design model implementationcheck check

check

Model-Based Testing

requirements design model implementation

test suitegenerator

test suite

check check

check

Model-Based Testing

textualrequirements

Simulink/Stateflow

formalisedproperties

generatedC code

implementationcheck

check

Model-Based Testing

textualrequirements

Simulink/Stateflow

formalisedproperties

generatedC code

implementation

test suitegenerator

test suite

check

check

Example: Cruise Control

OFF,0,FALSE

OFF,1,FALSE OFF,0,TRUE

ON,1,TRUE

DIS,2,TRUE DIS,0,TRUE


brake ∨ dec

brake ∨ dec

brake∨ dec

gas ∨ acc

gas ∨ accgas∨ acc

acc ∨ dec

gas ∨ acc

brake ∨ dec button button

button

button

buttonbutton

button

gas ∨ acc

gas ∨ acc

brakegas

∨ accbrake∨ dec

brake∨ dec

brake ∨ dec

Example: Generated C Code from Simulink

void i n i t ( s t a t e t ∗s ) {s−>mode = OFF ;s−>speed = 0 ;s−>enable = FALSE ;

}void compute ( i o t ∗ i , s t a t e t ∗s ) {

mode = s−>mode ;switch (mode) {

case ON: i f ( i−>gas | | i−>brake ) s−>mode=DIS ; break ;case DIS :

i f ( ( s−>speed==2 && ( i−>dec | | i−>brake ) ) | |( s−>speed==0 && ( i−>acc | | i−>gas ) ) )

s−>mode=ON;break ;

case OFF :i f ( s−>speed==0 && s−>enable && ( i−>gas | | i−>acc ) | |

s−>speed==1 && i−>button | |s−>speed==2 && s−>enable && ( i−>brake | | i−>dec ) )


}i f ( i−>button ) s−>enable = ! s−>enable ;i f ( ( i−>gas | | mode!=ON && i−>acc ) && s−>speed<2) s−>speed ++;i f ( ( i−>brake | | mode!=ON && i−>dec ) && s−>speed>0) s−>speed−−;

}

Example: Generated C Code from Simulink

void i n i t ( s t a t e t ∗s ) {s−>mode = OFF ;s−>speed = 0 ;s−>enable = FALSE ;

}void compute ( i o t ∗ i , s t a t e t ∗s ) {

mode = s−>mode ;switch (mode) {

case ON: i f ( i−>gas | | i−>brake ) s−>mode=DIS ; break ;case DIS :

i f ( ( s−>speed==2 && ( i−>dec | | i−>brake ) ) | |( s−>speed==0 && ( i−>acc | | i−>gas ) ) )


case OFF :i f ( s−>speed==0 && s−>enable && ( i−>gas | | i−>acc ) | |

s−>speed==1 && i−>button | |s−>speed==2 && s−>enable && ( i−>brake | | i−>dec ) )


}i f ( i−>button ) s−>enable = ! s−>enable ;i f ( ( i−>gas | | mode!=ON && i−>acc ) && s−>speed<2) s−>speed ++;i f ( ( i−>brake | | mode!=ON && i−>dec ) && s−>speed>0) s−>speed−−;

}

Formalised properties:

p1: G(

mode = ON ∧ speed = 1 ∧ dec ⇒ X(speed = 1))

p2: G(

mode = DIS ∧ speed = 2 ∧ dec ⇒ X(mode = ON))

p3: G(

mode = ON ∧ brake ⇒ X(mode = DIS))

p4: G(

mode = OFF ∧ speed = 2 ∧ ¬enable ∧ button ⇒ X enable)

Example

I = F

OFF,0,FALSE


ON,1,TRUE



brake ∨ dec

brake ∨ dec

brake∨ dec

gas ∨ acc


acc ∨ dec (p1)

gas ∨ acc


button

button

button (p4)button

button

button

gas ∨ acc

gas ∨ acc

brake (p3)gas

∨ accgasbrake

∨ decbrake∨ dec(p2)

brake ∨ dec

Preliminaries

Program:

State space Σ, input space Υ

Initial states I ⊆ Σ

Transition relation T ⊆ Σ×Υ× Σ

Preliminaries

Program:




Bounded Model Checking:

Check the existence of a path 〈s0, s1, . . . , sK 〉 of increasing lengthK from φ to φ′

φ(s0) ∧∧

1≤k≤K

T (sk−1, ik−1, sk) ∧ φ′(sK )

If SAT: satisfying assignment aka counterexample(s0, i0, s1, i1, . . . , sK−1, iK−1, sK )

Preliminaries

Program:




Bounded Model Checking:

Check the existence of a path 〈s0, s1, . . . , sK 〉 of increasing lengthK from φ to φ′

φ(s0) ∧∧

1≤k≤K

T (sk−1, ik−1, sk) ∧ φ′(sK )

If SAT: satisfying assignment aka counterexample(s0, i0, s1, i1, . . . , sK−1, iK−1, sK )

Test case generation:

φ = I and test goal φ′

Test case: input sequence 〈i0, . . . , iK−1〉, expected outcome

Chaining Test Cases

Temporal logic safety specification:

Set of properties, e.g., of type

G(mode = ON ∧ speed = 1 ∧ dec︸︷︷︸

assumption ϕ

⇒ X(speed = 1))

Chaining Test Cases




assumption ϕ

⇒ X(speed = 1))

Test goals: set of assumptions ϕ (finite paths)

Chaining Test Cases




assumption ϕ

⇒ X(speed = 1))


Test chain: from initial states I via all ϕs to final states F

Chaining Test Cases




assumption ϕ

⇒ X(speed = 1))


Test chain: from initial states I via all ϕs to final states F

Approach

1 Abstraction: property reachability graph

2 Optimisation: shortest path

3 Concretisation: compute concrete test case

Abstraction: Property Reachability Graph

Weighted, directed graph:

Nodes: test goals ϕ

Edges:

from I to all ϕsfrom all ϕs to F

pairwise links between ϕs

Edge weights: number of execution steps

Incrementally build graph by reachability queries:

ϕ1 ϕ3

I F

ϕ2 ϕ4




Edges:




Incrementally build graph by reachability queries: K = 1

ϕ1 ϕ3

I F

ϕ2 ϕ4

1

1

1




Edges:




Incrementally build graph by reachability queries: K = 2

ϕ1 ϕ3

I F

ϕ2 ϕ4

221

1

2

2

2

2 2

2

1

Existence of a Covering Path

Covering path: path that visits all nodes at least once.

There is a covering path from I to F

iff

(1) all nodes are reachable from I ,

(2) F is reachable from all nodes,and

(3) for all pairs of nodes (v1, v2),

(a) v2 is reachable from v1 or(b) v1 is reachable from v2.

ϕ1

I F

ϕ2

Reachability can be decided in constant time on the transitiveclosure of the graph.

Optimisation: Shortest Path Computation

Find a covering path from I to F :

Reduce to asymmetric travelling salesman problem (ATSP):

Tour that visits all nodes of a weighted directed graph exactlyonce

Transitive closure

ϕ1 ϕ3

I F

ϕ2 ϕ4

221

1

2

2

2

2 2

21





Transitive closure

ϕ1 ϕ3

I F

ϕ2 ϕ4

221

1

2

2

2

2 2

21

4 4

3

4

6

4 4





Transitive closure

ϕ1 ϕ3

I F

ϕ2 ϕ4

221

1

2

2

2

2 2

21

4 4

3

4

6

4 4

0





Transitive closure

ϕ1 ϕ3

I F

ϕ2 ϕ4

221

1

2

2

2

2 2

21

4 4

3

4

6

4 4

0

ATSP result: 〈ϕ2, ϕ3,F , I , ϕ4, ϕ1〉Shortest path: 〈I , ϕ4, ϕ1, ϕ2, ϕ3,F 〉

Concretisation: Computing the Test Chain

I2

−→ ϕ42

−→ ϕ12

−→ ϕ21

−→ ϕ32

−→ F

I (s0)∧T (s0, i0, s1) ∧ T (s1, i1, s2)∧ϕ4(s2, i2)∧T (s2, i2, s3) ∧ T (s3, i3, s4)∧ϕ1(s4, i4)∧T (s4, i4, s5) ∧ T (s5, i5, s6)∧ϕ2(s6, i6)

∧T (s6, i6, s7)∧ϕ3(s7, i7)∧T (s7, i7, s8) ∧ T (s8, i8, s9)∧ F (s9)

〈i0, . . . , i8〉 = 〈gas, acc, button, dec, dec, gas, dec, brake, button〉

Concretisation: Computing the Test Chain

I = F

OFF,0,FALSE


ON,1,TRUE



brake ∨ dec

brake ∨ dec

brake∨ dec

gas ∨ acc


acc ∨ dec (p1)

gas ∨ acc


button

button

button (p4)button

button

button

gas ∨ acc

gas ∨ acc

brake (p3)gas

∨ accgasbrake

∨ decbrake∨ dec(p2)

brake ∨ dec

Optimality

The test case chain is minimal if

(1) the program and the properties admit a test chain,

(2) all test goals are singleton sets, and

(3) the test chain visits each property once in the K -reachabilitygraph.

Optimality





ϕ1 ϕ3

ϕ2w1

w2

Optimality





ϕ1 ϕ3

ϕ2w1

w2

w < w1 + w2

Optimality





ϕ1 ϕ3

ϕ2w1

w2

w < w1 + w2

Reachability diameter d = length of maximum, shortest pathbetween any two states

There is a K ≤ d such that, under the preconditions (1) and (2),the test chain is minimal.

In practice, fix a bound K and obtain minimised chain.

Multi-State Test Goals

I ϕ1 ϕ2 F0 1

2

2

2

2

p1 : G(mode = OFF ∧ ¬enable ∧ button ⇒ X enable

)

p2 : G(mode = ON ∧ brake ⇒ X(mode = DIS)

)


I ϕ1 ϕ2 F0 1

2

2

2

2

Broken chain


I ϕ1 ϕ2 F

011

2

2

2

2

Broken chain

Path 〈I , ϕ1, ϕ2〉 not feasible in a single step, but requires twosteps.


I ϕ1 ϕ2 F1 1

2

2

2

2

Broken chain


Chain repair

Systematically increase edge weights of failed subpath

Minimality lost


I ϕ1 ϕ2 F1 1

2

2

2

2

Broken chain


Chain repair

Systematically increase edge weights of failed subpath

Minimality lost

Completeness

Succeeds if path admits chain in concrete program

If for each test goal the states are strongly connected

In practice: many systems are (almost) strongly connected.

General Case

Completeness

Not strongly connected systems:

Abstraction refinement

Abstraction Refinement

I ϕ1 ϕ2 F0 1

2

2

2

2


I ϕ1 ϕ2 F0 1

2

2

2

2

Abstraction refinement:

I ϕ1 ϕ2 F0 1

2

2

22

2

2


I ϕ1 ϕ2 F0 1

2

2

2

2

Abstraction refinement: Find any path

I ϕ1 ϕ2 F0 1

22

22

2

2


I ϕ1 ϕ2 F0 1

2

2

2

2

Abstraction refinement: Optimise with TSP solver

I ϕ1 ϕ2 F1

2

22

2

2

General Case

Completeness


Abstraction refinement

General Case

Completeness


Abstraction refinementMore general solver than TSP solver, e.g. ASP solver

General Case

Completeness



Multiple chains :

Partitioning by graph colouring

General Case

Completeness



Multiple chains :

Partitioning by graph colouring

Optimality

Would require to optimise over concrete system

In practice, minimised rather than minimal solutions relevant

Implementation

reactivesystemmodel

C codegenerator

staticanalyser

properties ChainCover test suite

Implementation

Properties specified as C functions:

void p 1 ( i o t ∗ i , s t a t e t ∗ s ) {CPROVER assume ( s−>mode==ON && s−>speed==1 && i−>dec ) ;

compute ( i , s ) ;a s s e r t ( s−>speed ==1) ;

}

Woven into program during test case generation.

BMC engine of Cbmc

Property reachability graph construction

Exploits incremental SAT solving

Chain repair by concrete chaining

Lkh travelling salesman problem solverClingo answer set programming solver

Benchmarks and Comparison

Benchmarks

Cruise control model

Window controller

Car alarm system

Elevator model

Robot arm model

Comparison with

FShell: a BMC-based test generator with test suiteminimisation

Random case generator with test suite minimisation

Klee: a test case generator based on symbolic execution

Results: Test Case Length

ut ut ut ut ut utut

utut

utut

ut

ut

* * * * * * * **

**

**

ld ld ld ldld

ldld

ld

ld

ld

ld

rs rsrs

rsrs

rs

rs

0

50

100

150

200

250

300

350

400

450

500

550

600

650

0 1 2 3 4 5 6 7 8 9 10 11 12

Number of benchmarks

Accu

mu

late

dte

stca

se

len

gth

s

� KLEE

♦ RANDOMTEST

∗ FSHELL

△ CHAINCOVER

Results: Test Case Generator Runtime

0

1000

2000

3000

4000

5000

0 1 2 3 4 5 6 7 8 9 10 11 12

ut ut ut ut ut ut ut ut ut utut

ut

* * * * **

**

*

*

*

*

ld ld ld ld ld ld ld ld ld ldrs rs rs rs rsrs

Accu

mu

late

dru

ntim

es

Number of benchmarks

� KLEE

♦ RANDOMTEST

∗ FSHELL

△ CHAINCOVER

Summary and Current Work

Summary

Test chain for reactive systems

Test goals from requirements, specification model, codecoverage criteria

Minimal test chain for single-state test goals, otherwiseheuristics

Experimental evaluation

Application: on-target testing, acceptance testing

Current work

Integrate acceleration to handle deep loops

Test chains for code coverage criteria, e.g. MC/DC

Further questions

Incremental test chain generation

In the case of model modificationsWhen test execution gets stuck due to a failed test goal

Download me!

http://www.cprover.org/chaincover

chaining test cases for reactive system testing test cases for reactive system testing peter...

Documents