software testing - uantwerpen · • state transition tables (state-to-state, event-to-state) •...

Universiteit Antwerpen

Software Testing

Chapter 2

2. Models


(Based on “Part II: Models” of Testing Object-Oriented Systems)

• Models • Why ? What ? How ?

• Combinational Models • Decision Tables: What ? How ? • Test Generation

• State Machines • What ? (variants: Mealy & Moore) • State Transition Tables (State-to-state, Event-to-state) • The FREE State Model (State ? Transition ?) • Test Generation (N+)

• Part III: Test Patterns

2. Models

!2


Testing Approaches

!3

(Smart) Poking Around

Brute Force

Systematic & Focused


Models are what distinguishes craft from engineering !

Testing is a search problem • trillions and trillions of possible input and state combinations

• only a few will reach, trigger and propagate bugs

Solutions ? • Brute force … does not work at this scale • Poking around … inefficient + false feeling of confidence.

• Model based = systematic, focused and automated • Systematic: we try every target combination (target ≈ model) • Focused: where are bugs likely ? (likely ≈ fault model) • Automated: more tests; more repeatable test (tools ≈ model)

Why Models ? (in Testing)

!4


What is a Model ? (in Testing)

TEST MODELS • subject

- e.g. implementation, component, … under test

• point of view/theory - required behaviour,

where are bugs likely ? • representation

- graphs (nodes & edges), - checklists

• technique - experience with

requirements, design, …

ENGINEERING MODELS • subject

- e.g. airplanes, buildings, weather, economics

• point of view/theory - e.g. gravitation, air-pressure,

law of demand and offer • representation

- wire frame image, blueprint - mathematical equations

• technique - skill and expertise of

modeller matters!5

coverage


(TEST) MODELS • Specify precisely

- what is allowed … what is NOT allowed

Different models representing various perspectives • necessarily

- complete - unambiguous - consistent - coherent

CARTOONS • sketching, refining,

documentation - initial analysis, design, …

• potentially - incomplete - ambiguous - inconsistent - not integrated

Cartoons vs. Models

!6


Scope: Testing, Validation, Checking, Verification

!7

meta-model

component representation

component implementation

required behaviour

observed behaviour

Consistency CheckingCompleteness

Checking

Responsability-

based Testing

Implementation-based Testing

Covered in this course !

Verification (checklists, proofs) [no execution]

Validation[Black box]

[White box]







2. Models

!8


Decision Table: example

!9

Condition Section Action Section

Variant # Claims # Insured Age

Premium Increase

Send Warning

Cancel

1 0 <= 25 50 No No

2 >= 26 25 No No

3 1 <= 25 100 Yes No

4 >= 26 50 No No

5 2 to 4 <= 25 400 Yes No

6 >= 26 200 Yes No

7 5 or more Any 0 No Yes

Implicit Assumptions • Logical operator to interpret a row ? (AND) • Can a person of 15 really be insured ? And what about 99 ?

Annual renewal of a hypothetical car insurance policy


Decision Table: alternative views

!10

Variant

1 2 3 4 5 6 7

Condition Section

Number of Claims

0 0 1 1 2-4 2-4 >= 5

Insured Age

<= 25

>= 26

<= 25

>= 26

<= 25

>= 26

Any

Action Section

Premium increase

50 25 100 50 400 200 0

Send warning

No No Yes No Yes Yes No

Cancel No No No No No No Yes

Column wise view

Decision Tree $50, no letter<= 25

Number of Claims

Age

Age

Age

Cancel

None

1

2-4

>= 5

$25, no letter>= 26

$100, send warning letter

$50, no letter

<= 25

>= 26



<= 25

>= 26

• easier to understand (and program)

• possibility to overlook a case


STEPS • 1) identify the decision

variables and conditions • n conditions ⇒ 2n variants

• 2) identify the resultant actions to be selected or controlled

• 3) identify which actions should be produced in response to particular combinations of conditions

• 4) derive the logic function • via truth table • validate completeness and

consistency

APPLICABLE ? • One of several distinct

responses is to be selected • … according to distinct cases of

input variables (finite !) • input cases are mutually

exclusive • response is independent of

• order of input variables • prior input or output

• prefer small tables (combinational explosion)

• typical for business rules, simple protocols (locks), …

Combinational Models: Basic Approach

!11 Universiteit Antwerpen

decision table with n conditions ⇒ 2n variants

usually fewer variants, only explicit variants are listed Implicit variants ? • don’t care: condition true or false, doesn’t change the action

- e.g.: when # claims >= 5, then age is “don’t care” condition • can’t happen: condition which is assumed never to become true

- e.g.: insured age < 16 or 18 (age when driving is allowed) - can sometimes occur when context changes (Ariane crash) - must be tested explicitly (sometimes is a “surprise”)

• don’t know: is an incomplete model - e.g. what happens with an age > 300 ?

can’t happen + don’t know (+ don’t care): typical source for bugs - specify resulting action anyway (defaults) - test for the result

Identify decision variable and conditions

!12


Deriving the logic function: truth table

!13

Logic function: only boolean variables as input and output • Expression consisting solely of AND, OR, XOR, NOT + conditions • Convert decision table to truth table; truth table into expression

Decision Variable Variant

Condition section

Condition 1 2 3 4 5 6 7

Number of claims 0 T T F F F F F1 F F T T F F F2-4 F F F F T T F

5+ F F F F F F T

Insured Age 25 or younger T F T F T F DC26 or older F T F T F T DC

Action section

Premium increase = 0 F F F F F F TPremium increase = 25 F T F F F F FPremium increase = 50 T F F T F F F

Premium increase = 100 F F T F F F F

Premium increase = 200 F F F F F T FPremium increase = 400 F F F F T F FSend warning F F T F T T FCancel F F F F F F T


Deriving the Logic Function: logic minimization

!14

Initial Development

Logic Minimization Technique +- upper limit, number of boolean variables

-- inspection 3

1950s KV matrix 5

1970s Cause-effect graph 8

1950-1960s Quine-McCluskey method 9

1960s Starner-Dietmeyer 14

1980s Exact minimization 20-25

1980s Heuristic minimization 20-25

• reduce logic function to compact but equivalent expression - fewer test cases are needed - faster testing (automated) - may reveal omissions or inconsistencies


• Incorrect value assigned to a decision variable • Incorrect or missing operator in a predicate • Incorrect or missing variable in a predicate • Incorrect structure in a predicate

- dangling else, misplaced semicolon, … • Incorrect or missing default case • Incorrect or missing action(s) • Extra actions • Structural error in decision table implementation

- falling through (i.e. a missing break in C++), incorrect value for an index or action selector

• Missing or incorrect class or method signature - when variants are implemented via polymorphism

• Generic errors - wrong version, incorrect or missing specification item, ambiguous

Decision Table: Fault Model

!15

No empirical models about

distribution of faults !


• All-Explicit Variants - Most intuitive approach - 1 test case for each explicit variant of all decision variables - example: 0 claims, 1 claim, 2-4 claims, 5+ claims,

25 or younger, 26 or older • Applicable

- decision tables with non-binary variables - weak for undefined “can’t happen” situations - weak when boundaries are undefined (⇒ “don’t know” situation)

⇒ Also apply non binary variable domain analysis

Test Generation: All-Explicit Variants

!16


Start from truth table (+ corresponding boolean function) • All-Variants: every variant in the table is tested once

- 2n possibilities: works only for smaller tables

• All-True: every variant that produces TRUE - equivalent to testing all minterms of logical function - inappropriate when important actions follow from false variants

• All-False: every variant that produces FALSE - equivalent to testing complement of all minterms of logical

function - inappropriate when important actions follow from true variants

• All-Primes: each prime implicant of boolean function at least once - i.e. each product term in a minimal sum-of-products form of

boolean expression All-true, all-false, all-primes may miss critical bugs !!

All-Variants, All-True, All-False, All-Primes


Each-Condition/All-Condition • each variable is made true once, with all other variables being false

+ one test case where all variables are true (and logic) + one test case where all variables are false (or logic)

Binary Decision Diagram Determinants • Starts from the truth table

- Produces a binary-decision diagram (see book p. 157-158) - Map the diagram into a determinant table (see book p. 159-160)

Variable Negation • Starts from sum-of-products boolean formula

- unique true points: … - near false points: …

Non-binary Variable Domain Analysis • domain testing (including boundary values) on decision variables

More test generation techniques

!18


Decision Table Test Suite Size

!19

Test Strategy Test size, Binary Decision Variables

Test Size, Non-binary Decision Variables

Test Power

All-Variants 2n 2n * 4 n 100%

Basic Variable Negation 0.1 * 2n 0.1 * 2n * 4n 97%

Each Condition/All- Conditions m + ∑ L(pi) (m + ∑ L(pi)) * 4n ?

All BDD Determinants d d * 4n ?

All-True t t * 4n ?

All-False f f * 4n ?

All-Primes m m * 4n ?

All-Explicit x x * 4n ?

For non-binary decision variable we have an expression (A <= x <= B) • test x = A - 1, x = A + 1, x = B -1, x = B + 1

• # tests may become smaller for ranges where x are adjacent


Inclusion Hierarchy

!20

All-Variants

All-FalseAll-True

All-Primes

Each-Condition/ All-Conditions

Basic VariableNegation

BDDDeterminants







2. Models


= system whose output is determined by both current and past input (with Decision tables, only current input influences result)

• identical inputs are not always accepted • identical inputs may produce different outputs

State machine consists of 4 building blocks • State: an abstraction that summarizes the past inputs • Transition: an allowable two-state sequence

• “accepting” and “resulting” state • Caused by an event; may result in an action

• Guarded transition: event + guard predicate • Event: an input (or an interval of time) • Action: the result or output that follows an event

Special states: initial state, current state, final state

What is a state machine ?

!22


MOORE State Machine • transitions are passive • states are active

- may have output action - every output action has at

least one state

MEALY State Machine • transitions are active

- may have output action - any output action may be

used in more than one transition

• states are passive

Variants: Mealy and Moore

!23

• Mathematically equivalent • MEALY is preferred for engineering design • UML allows both !

- Hybrids are bad for test design !


Two-player Squash: Mealy State Machine

!24

game started

player1 served player2 served

player1 won player2 won

p2_Start()/ simulateVolley()

p1_Start()/ simulateVolley()

p1_WinsVolley() [this.p1.score()< 20] / this.p1AddPoint() simulateVolley()

p2_WinsVolley() [this.p2.score()< 20] / this.p2AddPoint() simulateVolley()p1_WinsVolley()/

simulateVolley()

p2_WinsVolley()/ simulateVolley()

p1_WinsVolley() [this.p1.score()== 20] / this.p1AddPoint()

p2_WinsVolley() [this.p2.score()== 20] / this.p2AddPoint()

p1_IsWinner()/ return TRUE

p2_IsWinner()/ return TRUE


Two-player Squash: Moore State Machine

!25

game started

Player1 serving simulateVolley()

Player1 won p1_AddPoint()

Player2 won p2_AddPoint()

p2_Start()p1_Start()

Player2 serving simulateVolley()

p1_WinsVolley()

p2_WinsVolley()

Player1 continues p1_AddPoint() simulateVolley()

Player2 continues p2_AddPoint() simulateVolley()

Answering Player2 won return TRUE;

Answering Player1 won return TRUE;

p1_IsWinner()p1_IsWinner() p2_IsWinner()p2_IsWinner()

p1_WinsVolley() [this.p1_Score() < 20] p1_WinsVolley()

[this.p1_Score() == 20]

p2_WinsVolley() […]

p2_WinsVolley() [this.p2_Score() == 20]

p1_WinsVolley() p2_WinsVolley()p2_WinsVolley()

p1_WinsVolley()


state-to-state format • row = accepting state; column = resulting state • cell = transition = event (with guard) + action

event-to-state format • row = event (with guard); column = accepting state • cell = transition = action + resultant state

expanded state-to-state format • 2 state-to-state tables: 1 for events and 1 for actions • separates the transition function and output function

State Transition Tables

!26


Example: State-to-state

!27

CurrentState

Resultant State/Event/Action

Game Started

Player 1 Served Player 2 Served Player 1 Won Player 2 Won

Game Started

p1_start() p2_start()

simulateVolley() simulateVolley()

Player 1 served

p1_winsVolley() [p1 score < 20]

p2_winsVolley() p1_winsVolley() [p1 score == 20]

this.p1AddPoint(), SimulateVolley()

simulateVolley()

Player 2 served

p1_winsVolley() p2_winsVolley() [p2 score < 20]

p2_winsVolley() [p2 score == 20]

simulateVolley() this.p2AddPoint(), SimulateVolley()

Player 1 won

p1_IsWinner()

return TRUE

Player 2 won

p2_IsWinner()

return TRUE


Example: Event-to-state

!28

Current State/Action/Next State

Event Guard Game Started Player 1 Served Player 2 Served Player 1 Won Player 2 Won

p1_start() simulateVolley()

Player 1 Served

p2_start() simulateVolley()

Player 2 Served

p2_WinsVolley()

DC (don’t care)

simulateVolley()

Player 2 Served

p2_Score < 20

this.p2AddPoint() ,simulateVolley()

Player 2 Served

p2_Score == 20

this.p2AddPoint()

Player 2 Served

p1_WinsVolley()

… …

p1_IsWinner()

return TRUE

Player 1 Won

… …


Flattened Regular Expression (FREE) • class under test is flattened

• all inherited features are collapsed in class under test • assume “type-substitution” principle • instance of a class may be substituted by an instance of a subclass

• model behaviour as state machine

• restrictions to UML statechart notation • few changes to UML statechart notation (e.g., alpha/omega states)

• can be expressed in UML

• results in testable model • no ambiguities (among others precise definition of “state”) • automatic generation of test cases

The FREE State Model


• A particular subset of a class combinational value set • Represented by a State Invariant (predicate) Class Account { private: AccountNumber number; Money balance; Date lastUpdate}

What is State ?

!30

Account.balance

Account.lastUpdate

Account.number

Open StateInactive State3.650.182.500.000

“states”


• state invariant defines a subset of legal values for a class • state invariant is same or stronger than class invariant • method postconditions + state invariants = all states

• don’t use hybrids (i.e. no mixing of Mealy and Moore)

• alpha state • null state: represent declared object before construction • accept only constructor, new, … (alpha transition) • is NOT the same as the UML initial state (solid circle)

• omega state • null state: reached after deletion, destruction, out of scope • accepted explicit or implicit destructors (omega transition) • is NOT the same as the UML final state (bulls-eye)

Implications

!31

alpha-omega cycle


Transition = • 1 state invariant for accepting state • 1 state invariant for resulting state • 1 associated event

• message sent to the class under test • interrupt or similar external control (timer, …)

• [optional guard expression] • predicate expression evaluated in the context of class under test

• 0 or more actions • message sent to a server object of the class under test • response provided by an object of the class under test

What is a Transition ?

!32


What should happen when a state machine in a given state receives an event not specified for this state ? • ignore

• standard semantics for state machines • unacceptable for testing purposes

• omitted • incomplete specification: extend the state machine

• illegal (or “impossible”) • illegal event = valid event, not acceptable for the current state

(e.g. “pop()” on an empty stack) • if accepted results in an illegal transition

Sneak path • beware: guarded transitions, what if the guard is FALSE ? • transition to an “Illegal event exception” state

Unspecified Event/State Pairs


Response Matrix = Modified event-to-state table • ROWS: list events + guards

• unguarded event: 1 row • guarded event: 1 row for each unique event/guard combination

• one row for each truth combination of subexpressions + additional column for all sub expressions

• if event is sometimes guarded, then include a “don’t care” row • COLUMNS: list Accepting states • CELLS: list responses

• error codes for possible responses

Response Matrix

!34


0. Accept (perform specified transition) 1. Queue (queue event for subsequent evaluation and ignore) 2. Ignore 3. Flag (return a non zero error code) 4. Reject (raise an exception) 5. Mute (disable the source of events and ignore) 6. Abend (invoke abnormal termination and halt process)

Response Matrix: Possible Responses


Example: Response Matrix

!36

Accepting State/Expected Response

Events Guards alpha A B … C omega

constructor --- --- --- 0 6 6 6 6 6

event 1 x == 0

DC --- --- --- --- 1 … 2 6

F --- --- --- 0 --- … --- ---

T --- --- --- 0 --- … --- ---

event 2 i > 10 k == max isReset()

DC DC DC --- 2 5 … --- 6

F F F --- --- --- … 5 ---

F F T --- --- --- … 0 ---

F T F --- --- --- … 0 ---

… … … …

destructor --- --- --- --- 0 0 0 0 2

if event is sometimes guarded, then include a “don’t care”


s states, e events, a actions gives (s x a) es possible implementations • 5 states, 2 events and 2 actions: 10 billion possibilities • only one correct implementation

Possible faults • missing or incorrect transition • missing or incorrect event • missing or incorrect action • an extra, missing or corrupt state • a sneak path (a message is accepted when it shouldn’t be) • an illegal message failure (unexpected message causes a failure) • trap door (implementation accepts undefined messages)

State Based Testing: Fault Model

!37

No empirical models about

distribution of faults !


• structure check-list (Table 7.5) • structurally complete and consistent

• state name check-list (Table 7.6) • avoid weak, ambiguous or inappropriate state names

• guarded transition check-list (Table 7.7) • check logic and structure of guard expressions

• flattened machine check-list (Table 7.8) • free of level-to-level conflicts • consistent behaviour for inheritance tree • reuse of superclass test suites

• robustness check-list (Table 7.9) • safe and correct behaviour under failure modes

Verify the State Model

!38


N+ Test Generation • Features

• exercise all implicit transitions to reveal sneaky paths • implementation must be able to report current state

state reporter methods (available ? encapsulation ?) • Steps

• Develop a FREE model + Derive response matrix • Generate round-trip test cases via Transition Tree (see p. 249)

cover all branches (= transitions) at least once identify parameter values, expected state and exception

• Generate sneak path test cases cover all branches with unspecified transitions verify response + resulting state

• Sensitize the transitions find input values for all guards

Test Generation


Based on years of experience with state-based testing of hardware and telecommunications infrastructure

• All Transitions • every specified transition at least once

• All n-Transition sequences • every specified transition sequence of n events

• All round-trip Paths • every sequence of specified transitions beginning and ending in

the same state is exercised at least once • M-Length Signature

(when state reporter methods are absent) • signature = sequence of output actions unique for given state • verify whether we reached the given state by verifying signature

State Machine Coverage

!40







2. Models


pattern = • generalized solution to a specific recurring (design) problem

• capture the essence of proven solutions

pattern template • includes name (design discussions) • context

• when to apply, when NOT to apply • forces: different interests pull towards various directions

• trade-offs • advantages and disadvantages in terms of forces

• known uses

(Test) Patterns

!42


• name • intent • context (incl. test scope) • fault model

• what kind of faults are the target ? • motivate: why reach faults + trigger and propagate failures

• strategy • how to design and implement the test suite • procedure, oracle, automation • example

• entry criteria • preconditions for using this pattern • when is the“Implementation Under Test” (IUT) test ready ?

• exit criteria • defines the results necessary to achieve an adequate test

Test Pattern Template


(See also IEEE standard 829) • prepare a test plan

• what to test when • deliverables • stop criteria

• design the test suite • test suite hierarchy (i.e. V-model) • procedures for running the tests • prepare test cases (using appropriate test pattern)

• test the component • results for each test case: pass / no pass / unable to execute • store results (and follow-up action) for later reference

• prepare test summary report • Evaluate the operability of the IUT

Test Documentation

!44




• State Machines • What ? (variants: Mealy & Moore) • State Transition Tables (State-to-state, Event-to-state) • The FREE State Model (State ? Transition ?)

alpha-omega cycle • Test Generation (N+)


Conclusion


Patterns are NOT a good vehicle for • Analytical comparison (and contrasting) • Presenting a general model, theory, point of view • Defining terms + conceptual relationship • Lengthy and complex system of ideas

I want you to teach some of the material & serve as opponent for your colleagues

• Will force you to acquire deep understanding • In depth understanding of parts

is preferred over broad understanding of the whole

Patterns are not a Panacea

!46


Preparing slides for a 2-hour lecture takes 8 hours ! • scope of the lecture is known

• message (what do I want the audience to remember ?) • story (how will I sequence the material from known to unknown ?)

Avoid “death by bullet points” • use tables / schemas / animation / … • slides are a tool (not a goal in themselves)

Passionate lecturing • use personal anecdotes, examples, …

• Examples? Industrial Guest Lectures! • relate to your audience

Tips

!47

software testing - uantwerpen · • state transition tables (state-to-state, event-to-state) •...

Documents