25/06/2015marius mikucionis, aau sse1/22 principles and methods of testing finite state machines –...

04/18/23 Marius Mikucionis, AAU SSE 1/22

Principles and Methods of Testing Finite State

Machines – A Survey

David Lee, Senior Member, IEEE

and

Mihalis Yannakakis

invited paper

Part about Conformance Testing


Overview• Main concepts: FSM, sequences, equivalence• Conformance testing problem and assumptions• Separating family of sequences• Status messages and reset• Distinguishing sequences• Identifying sequences• Polynomial time randomised algorithm• Heuristic procedures and optimisations• Extensions: one machine, more states, partially

specified


Concepts: FSMFinite State Machine (Mealy machine):• Is a tuple: M=(I, O, S, , ):

– I, O – input and output symbol sets (alphabets),

– S – state set, : SI S – transition function, : SI O – output function.

• Example:– When the machine is in state sS and

receives input symbol aI it moves to state (s, a) and produces output symbol (s, a).


Concepts: sequences and state equivalence

• We extend transition and output functions:– Let x be an input string: x=a1,…,ak, aiI, for i=1..k

– then (s1, x)=sk+1 where si+1=(si, ai), for i=1..k

– and (s1, x)=b1…bn where bi=(si, x), for i=1..k

• x – is a sequence and b1…bn is a response to it.

• Equivalent si, sj S xI*: (si, x) =(sj, x)

• If si and sj are not equivalent then: separating sequence xI*: (si, x) (sj, x)


Machine equivalence

• Equivalent A, B FSM sASA sB SB: sB equivalent to sA

sBSB sA SA: sA equivalent to sB

• There are many equivalent machines• Each equivalent machine class

contains minimized machine M with minimum number of states (all states are unique).

• Minimized machine is unique.


Conformance testing problem• Given:

– Complete information of specification machine A (states, transition and output function)

– Implementation machine B, black box, only I/O is observable

• Goal:– Determine whether B is correct implementation

of (conforms to, is equivalent to) A by applying a test sequence to B and observing the output.

• Checking sequence for machine A with n states is an input sequence x that distinguishes A from any non-equivalent machine B with n states: (sB, x) (sA, x) B is not equivalent to A


Conformance testing assumption

• Specification A is strongly connected– It must be possible to reach all states

• A is reduced (minimized)– We can determine equivalence only to

minimized machine, since equivalent states are not distinguishable.

• B does not change during experiment and has the same input alphabet as A

• B has no more states than A– Assume, faults do not increase number of

states, only:• Wrong output on transition• Wrong state in transition destination


Conformance test structure• Algorithm structure:

– Initialization: move to some known state s1:• If s1 is given verify it (not always possible) • Else apply homing sequence that takes to some known

state s1 (possible for minimised machines)– Verify similarity of B to A– Verify each transition (si, a)=sj :

• Apply sequence that moves machine to si

• Apply a• Verify that machine is in sj

• Methods:– Status and Reset messages– Distinguishing sequences– Identifying sequences– Randomised sequences


Separating family of sequences

• Separating family of sequences for A is a collection of n sets i (one set for each state):– For every pair of states si, sj (sisj) there is

sequence :1) (si, ) (sj, )

2) is a prefix in some xii and a prefix in some xjj

i is called a separating set of state si

• Elements of i are separating sequences of state si

• Examples:– A has a preset distinguishing sequence x then all

sets i may be equal to {x}.– UIO sequences fulfil 1) but may violate 2).


Separating family properties

• General construction algorithm for reduced FSMs:– If A is reduced si,sj x – separating sequence– Partition states into blocks based on (sk, x):

• For every sk put x into Zk

– Repeat procedure for each block until all blocks become singletons.

• Every pair of states has a separating sequence with common prefix.

• Zi contains <n-1 sequences of length n.• Key property: only one state at most gives

the same response to all elements of particular i


State and machine similarity

• State similarity: – qiSB is similar to siSA (qi, x)=(si, x) xi

• Because of key property: qi can be similar to at most one state of A

• Machine similarity:– B is similar to A siSA qiSB qi is similar to si

• All qi are distinct and B has at most n states one-to-one correspondence between A and B states.


Status messages and Reset

• FSM has a reset capability if special input r takes the machine from any state to initial s1.

• If r input is defined for all states of B then reset is reliable.

• Status message tells the current state of machine without changing it.

• Reliable status message guarantees that state will stay the same as before message.

• Unreliable status messages must be applied twice when the state is expected to change.

s

status/[state]

s1r/

r/r/

r/


Test with reliable reset• Let i be a family of separating sets• Build a spanning tree with states in nodes

from diagram of machine A• B similarity to A check:

– For every siSA • for every xZi:

– Reset B to state s1 by applying r– Move to state si according to tree path from s1 to si

– Apply x we are sure that B moved to a state similar to s i

we are sure that B has states similar to A ones, B has no more states than A has, B is similar to

A


Test with reliable reset (2)

• Check all other transitions (si, a)=sj :– For every xZj:

• Reset to s1 by r• Move to state si according to tree path from

s1 to si

• Apply a• Apply x

we are sure that transition (si, a)=sj is OK

B is isomorphic to A


Test with distinguishing sequences

• Let A have an adaptive distinguishing sequence, then Zi={xi}, where xi is a path in decision tree from root to state si

(si, sj) is a pre-computed transition sequence that takes machine from state si to state sj

• ti= (si, xi) – some state after separation• Similarity test:

– x1 (t1, s2)x2 (t2, s3)x3 …xn (tn, s1)x1

we have visited all n states and observed all distinguishing responses to distinguishing sequences

• Transition (si, a)=sj test when in state tk: (tk, si-1)xi-1(ti-1, si) a xj


Test with identifying sequences

• Problem: to verify state si similarity we need to apply separating sequence |Zi| times precisely on the same state.

• Example:– Separating family: Zi={a, b}

– Machine is in state s1

– Apply “aaa”, observe “000”

– B was in q0, moved to q1, q2, q3

– However, at least two of states are the same q3=qi for some i=0..2 we have already applied a

on q3 let’s apply b on it! we have successfully applied a and b on the same

state, B has state q3 similar to state s1 in A.

1

2 3

a/0

a/0a/1

b/1

b/1

b/0


Test with identifying sequences (2)

• A has n states• Similarity to si with separating set Zi={z1, z2}:

– Let qr=(si, z1(ti, si))r ), where ti=(si, z1)– Apply qn

– Then r<n : qr= qn, i.e. we have already applied z1 successfully on qn

– Apply z2

sequence (z1(ti, si))nz2 identifies a state in B which is similar to si

• Transition check: reuse reliable reset idea, because identifying sequences actually reset the state.


Polynomial time randomized test

• Similarity:– For i=1,…,n:

• Repeat ki times:

– Apply sequence that takes to si from current state

– Choose a separating sequence zZi uniformly at random

– Apply z

• Let x be random input formed sequence from similarity test with– ki=O(n |Zi| min(p, |Zi|) log n)

• Then it can be shown that B is similar to A with high probability


Heuristic procedures and optimizations

• Checking sequences guarantee complete fault coverage, but sometimes they are too long.

• Success example: circuit testing is based on faults model significantly limit possible faults.

• Covering paths:– Transition checking with UIO sequences leads to

Postman Tour Problem NP-hard.– Random walk may be “trapped” if system has “narrow

passages” or has just few faults exponential.– Guided random walks records partial history and

makes random choice based on priorities from history.– Test sequences from combinations of sub-sequences:

overlap in sub-sequences.


Summary of algorithmsComplexity is very sensitive to preliminary conditions and requirements to the system:

p=|I| – number of inputs, n=|S| - number of states

Sequences Length Time

Reliable reset pn3 pn3

Distinguishing pn3 pn3

Identifying Exponential Exponential

Randomized Polynomial Polynomial


Conformance test extensions

• One black-box only:– No difference for deterministic test algorithms– Randomized algorithms are more effective on a

single fixed machine: fault probability is squared while test length is doubled

• More states than in specification:– Unknown states and transitions have “combination

lock” property that require exponentially long sequences to cover.

• Partially specified machines:– Special treatment for undefined transitions may

transform machine to fully specified machines.– One fault - exponentially many machines– Only randomised checking has polynomial

complexity


Evaluation and critics• Valuable digest of completely solved

problems.• Good introduction to partially solved ones.• Easy to read: concepts are presented

following the pattern: motivation, definition and simple example.

• Most of algorithms are “encoded” in complex sentences instead of nested blocks – gives motivation, but looses clarity.


Thank you for your attention!

Questions?


Interesting proposition

Let A and B satisfy the assumptions, then the following are equivalent:

1) A and B are isomorphic (there exists a bijection between equivalent A and B states)

2) A and B are equivalent3) At least one state of A has an

equivalent state in B

25/06/2015marius mikucionis, aau sse1/22 principles and methods of testing finite state machines –...

Documents