april 20, 2006 model program based black-box testing, lentedagen, vught, the netherlands 1 model...

April 20, 2006April 20, 2006 Model Program Based Black-Box Testing, LModel Program Based Black-Box Testing, Lentedagen, Vught, The Netherlandsentedagen, Vught, The Netherlands

11

Model Program Based Model Program Based Black-Box TestingBlack-Box Testing

Margus VeanesMargus Veanes

Foundations of Software Foundations of Software Engineering Microsoft Research, Engineering Microsoft Research,

RedmondRedmond

April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

2

System-level black-box System-level black-box testingtesting

►Requires model of system behaviorRequires model of system behavior►Behavior is often Behavior is often

reactive/nondeterministicreactive/nondeterministic Implementation is multi-threaded or Implementation is multi-threaded or

distributeddistributed Thread scheduling hard to controlThread scheduling hard to control

►State space is typically infiniteState space is typically infinite Objects, unbounded value typesObjects, unbounded value types

►Traditional FSM-based testing does often Traditional FSM-based testing does often not scale not scale


3

Model based testingModel based testing

Model

Implementation

Test OracleTest Cases

Are run by Providesactual results for

PassFail

Provides expected results forGenerates

UserInfo


4

Our approachOur approach

► Behavior is described by Behavior is described by model programsmodel programs Written in Written in Spec# Spec# oror AsmL AsmL Describe reactive system behavior as Describe reactive system behavior as model model

automataautomata Use Use alternating simulation alternating simulation for conformancefor conformance

► Model exploration, validation and model-Model exploration, validation and model-based testing are provided by the based testing are provided by the Spec Spec ExplorerExplorer tool developed at MSR tool developed at MSR Supports scenario controlSupports scenario control Provides Provides offlineoffline as well as as well as onlineonline testing testing Views test cases as Views test cases as game strategiesgame strategies Checks conformance violationsChecks conformance violations


5

The big pictureThe big picture

Model program

Constrained model

Test cases

Pass/Fail

Implementation

Scenario control

Modeling

Test generation

Test execution Coding

Online testing


6

Conformance checkingConformance checkingcomponent viewcomponent view

Constrained model

Model

IUT wrapper

IUT

Refinement checker

Controllable

actions

Controllable

actions

Observable

actions

Observable

actions

Pass/Fail

d1……

c1

Object mapping


7

Example: Example: ChatChat

IUT wrapper

IUT

Chat server

Controllable actions:

Observable actions:

Login …Send

Receive

Refinement checker

Constrained chat model

b.Send(“hi”)

m.Send(“hello”)

b.Send(“bye”)

m.Rcv(“hi”)

m.Rcv(“bye”)

b.Rcv(“hello”)

b.Rcv(“hello”)

m.Rcv(“bye”)

m.Rcv(“hi”)

b.Rcv(“hello”)

BobMarym

b

Object mapping


8

Refinement relationRefinement relation

►Conformance between the model and the Conformance between the model and the system under test is system under test is alternating refinementalternating refinement

b.Send(“hi”)

m.Send(“hello”)

Spec System

b.Send(“hi”)

m.Send(“hello”)

m.Receive(b,“hi”)

b.Receive(m,“hello”)


m.Receive(b,“hi”)



9

Example: Example: Valid chat scenarioValid chat scenario

Bob Server Mary

B.Send(“hi”)M.Send(“hello”)

M.Receive(B,“hi”)

B.Receive(M,“hello”)

Tim

e

Service should guarantee Service should guarantee local consistencylocal consistency of message delivery (FIFO of message delivery (FIFO wrt sender)wrt sender)

SendSend is a is a controllable controllable actionaction

ReceiveReceive is an is an observable observable actionaction

B.Send(“bye”)

M.Receive(B,“bye”)


10

Model programs and Model programs and automataautomata

► States are mappings of variables to values States are mappings of variables to values first-order structuresfirst-order structures

► Initial state is given by initial assignment to Initial state is given by initial assignment to variables variables

► Actions are defined by method invocationsActions are defined by method invocations Actions are either Actions are either controllablecontrollable or or observableobservable

► An action An action mm((vv)) is is enabledenabled is a state is a state SS if the if the preconditionprecondition of of mm((vv)) is true in is true in SS

► An action An action mm((vv)) transitionstransitions from state from state SS11 to to statestate SS22 if if mm((vv)) is enabled in is enabled in SS11 and its and its execution in execution in SS11 yields yields SS22


11

Model program : Model program : Chat serviceChat service

State:

Actions:

class Client { bool entered = false; Map<Client, Seq<string>> unreceivedMsgs = Map{};}

class Client { void Send(string message) requires entered; { foreach (Client c in enumof(Client), c != this, c.entered) c.unreceivedMsgs[this] += Seq{message}; }

void Receive(Client sender, string message) requires sender != this && unreceivedMsgs[sender].Length > 0 && unreceivedMsgs[sender].Head == message; { unreceivedMsgs[sender] = unreceivedMsgs[sender].Tail;}…}

ControllableControllable::

Observable:Observable:


12

Online testingOnline testing

►For large reactive systems deriving an For large reactive systems deriving an exhaustive test-suite is infeasible exhaustive test-suite is infeasible

►Merge test generation and test Merge test generation and test execution into one processexecution into one process

►Online testing is a form of model-based Online testing is a form of model-based stress testingstress testing

►Purely random exploration is wastefulPurely random exploration is wasteful Use Use scenario controlscenario control to direct online testing to direct online testing


13

Exploration & Scenario Exploration & Scenario controlcontrol

► Exploration is the process of unfolding a rule-Exploration is the process of unfolding a rule-based model program into a transition system based model program into a transition system Actions move the system from state to stateActions move the system from state to state State comes from variables containing valuesState comes from variables containing values Compound data types are possible (sets, maps, Compound data types are possible (sets, maps,

sequences, etc…)sequences, etc…) Objects are identitiesObjects are identities

► Scenario control Scenario control Imposes limits on explorationImposes limits on exploration

► Using filters, state groupings, Using filters, state groupings, scenario actionsscenario actions , … , …


14

Example Demo: Example Demo: exploration of a coffee machineexploration of a coffee machine


15

Scenario controlScenario control

► In general the transition system is infinite, but we In general the transition system is infinite, but we can impose limitscan impose limits

► GoalGoal Create a state space of manageable size that satisfies a Create a state space of manageable size that satisfies a

given testing goalgiven testing goal► Two main tasksTwo main tasks

Restrict action parameter domains to interesting valuesRestrict action parameter domains to interesting values Restrict the state space to interesting statesRestrict the state space to interesting states

► NoteNote: : The two tasks are not necessarily independent!The two tasks are not necessarily independent!


16

Scenario control (cont.)Scenario control (cont.)

► The following techniques are used in Spec The following techniques are used in Spec ExplorerExplorer State filtersState filters Stopping conditionsStopping conditions State groupingsState groupings Search orderSearch order Enabling conditions on actionsEnabling conditions on actions Scenario actionsScenario actions

► Usually a combination of all (or some) of the Usually a combination of all (or some) of the techniques is needed to achieve desired effecttechniques is needed to achieve desired effect


17

Example of scenario actions: Example of scenario actions: Chat initializationChat initialization

► Use a parameterized scenario action Start.Use a parameterized scenario action Start. Invoking Start(Invoking Start(nn) produces a sequence of ) produces a sequence of nn Create()Create() invocations invocations

followed by followed by nn Enter()Enter() invocations invocations[Action(Kind=Scenario)]void Start(int nrOfMembers) requires enumof(Client) == Set{}; { Seq<Client> clients = Seq{}; // 1-- The given number of // clients are created for (int i = 0; i < nrOfMembers; i++) clients += Seq{Create()};

// 2-- all clients enter the session for (int i = 0; i < nrOfMembers; i++) clients[i].Enter();}

Start(2)

Create()/c0;Create()/c1;c0.Enter();c1.Enter();


18

Test generationTest generation► The result of exploration is a finite transition system The result of exploration is a finite transition system

with observable and controllable actions : with observable and controllable actions : a model a model automaton A. automaton A.

► Testing can be viewed as a Testing can be viewed as a gamegame between two players between two players Testing tool (TT) Testing tool (TT)

► Making Making controllablecontrollable transitions transitions Implementation under test (IUT)Implementation under test (IUT)

► Making Making observableobservable transitions transitions► The “rules of the game” are dictated by The “rules of the game” are dictated by AA► Tests are generated fromTests are generated from A. A. Test cases are Test cases are strategiesstrategies

A strategy tells what move TT should make in a state where A strategy tells what move TT should make in a state where its actions are enabledits actions are enabled

The purpose of a strategy may be to achieve transition The purpose of a strategy may be to achieve transition coverage or to reach a certain goal statecoverage or to reach a certain goal state


19

Demo: chatDemo: chat


20

ExperiencesExperiences

► Most tool features driven by demands of Most tool features driven by demands of internal users at MS (mostly testers)internal users at MS (mostly testers)

► Models help discover more bugs during Models help discover more bugs during modeling (design bugs) than testingmodeling (design bugs) than testing Testers do not get credit for finding those Testers do not get credit for finding those

bugs !!!bugs !!!

► During testing models help discover deep During testing models help discover deep system-level bugs where manual test system-level bugs where manual test scenarios don’tscenarios don’t Such bugs are hard to understand and fixSuch bugs are hard to understand and fix


21

Experiences (cont.)Experiences (cont.)

► Bugs appear in both models and Bugs appear in both models and implementations implementations (the ratio is roughly 50-50)(the ratio is roughly 50-50)

► Code coverage is a poor measure for testing Code coverage is a poor measure for testing concurrent software, often a single execution concurrent software, often a single execution thread provides the same coverage.thread provides the same coverage.

► New versions of implementation usually New versions of implementation usually require only local changes to models, require only local changes to models, whereas manual tests often need to be whereas manual tests often need to be rewritten completelyrewritten completely

► The tool is built on .NET but has been used to The tool is built on .NET but has been used to test distributed C++ applications. Most test test distributed C++ applications. Most test harnesses at MS are built in managed code.harnesses at MS are built in managed code.


22

OutlookOutlook

► User wish list:User wish list: Language independence for modelingLanguage independence for modeling

► Use C#, VB, …Use C#, VB, … Better control over explorationBetter control over exploration

► Scenario-oriented modelsScenario-oriented models Model compositionModel composition

► Model features separately and compose the modelsModel features separately and compose the models Model validationModel validation

► Validate properties on modelsValidate properties on models

Drives the focus of current research at Drives the focus of current research at FSEFSE


23

Related foundational workRelated foundational work

►Abstract state machinesAbstract state machines► Interface automataInterface automata► IInput-nput-OOutput utput COCOnformance (nformance (IOCOIOCO) )

theorytheory►MMarkov arkov DDecision ecision PProcess (rocess (MDPMDP) theory) theory


24

Thank you!Thank you!

►FSE web site:FSE web site: http://http://research.microsoft.comresearch.microsoft.com/foundations/foundations

►Downloading Spec Explorer:Downloading Spec Explorer: http://http://

research.microsoft.com/specexplorerresearch.microsoft.com/specexplorer

Questions?Questions?

april 20, 2006 model program based black-box testing, lentedagen, vught, the netherlands 1 model...

Documents

modelbased testing

model automata

model programs

chat model b

conformance model exploration

systemlevel blackbox

sendhello b

hello slide