model-based testing
DESCRIPTION
Model-Based Testing. Ed Brinksma. University of Twente Dept. of Computer Science Formal Methods & Tools group Enschede The Netherlands. ARTIST2 Summer School Nässlingen. Contents. introduction & background testing pre-orders input/output & quiescence ioco implementation relation - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/1.jpg)
Model-Based Testing
Ed Brinksma
University of TwenteDept. of Computer Science
Formal Methods & Tools group
EnschedeThe Netherlands
ARTIST2 Summer School
Nässlingen
![Page 2: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/2.jpg)
October 1, 2005 ARTIST2 Summer School 2
Contents
introduction & background testing pre-orders input/output & quiescence ioco implementation relation test generation TorX test case study real-time testing
![Page 3: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/3.jpg)
October 1, 2005 ARTIST2 Summer School 3
Contents
introduction & background testing pre-orders input/output & quiescence ioco implementation relation test generation TorX test case study real-time testing
![Page 4: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/4.jpg)
October 1, 2005 ARTIST2 Summer School 4
Practical problems of testing
Testing is: important much practiced 30% - 50% of project effort expensive time critical not constructive
(but sadistic?)
But also: ad-hoc, manual, error-prone hardly theory / research no attention in curricula not cool :
“if you’re a bad programmer you might be a tester”
Attitude is changing: more awareness more professionalImprovements possible
with formal methods !?
![Page 5: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/5.jpg)
October 1, 2005 ARTIST2 Summer School 5
Types of Testing
unit
integration
system
performancerobustness
functional behaviour
white box
black box
Level
Accessibility
Aspect
usability
reliability
![Page 6: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/6.jpg)
October 1, 2005 ARTIST2 Summer School 6
Test Automation
Traditional test automation= tools to execute and manage test cases
specification
testtool
implementationunder test
pass
fail
TTCNTTCNtestcase
s
Why not generatetest automatically?!
![Page 7: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/7.jpg)
October 1, 2005 ARTIST2 Summer School 7
formal world
concrete world
Verification is only as good as the validity of the model
on which it is based
Verification is only as good as the validity of the model
on which it is based
Verification and Testing
Verification : formal manipulation prove properties performed on model
Testing : experimentation show error concrete system
Testing can only show the presence of errors, not their
absence
Testing can only show the presence of errors, not their
absence
![Page 8: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/8.jpg)
October 1, 2005 ARTIST2 Summer School 8
Testing with Formal Methods
Testing with respect to a formal specification
Precise, formal definition of correctness :
good and unambiguous basis for testing
Formal validation of tests
Algorithmic derivation of tests :
tools for automatic test generation
Allows to define measures expressing coverage
and quality of testing
![Page 9: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/9.jpg)
October 1, 2005 ARTIST2 Summer School 9
Challenges of Testing Theory
Infinity of testing: too many possible input combinations -- infinite
breadth too many possible input sequences -- infinite depth too many invalid and unexpected inputs
Exhaustive testing never possible: when to stop testing ? how to invent effective and efficient test cases with
high probability of detecting errors ?
Optimization problem of testing yield and invested effort usually stop when time is over ......
![Page 10: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/10.jpg)
October 1, 2005 ARTIST2 Summer School 10
test
executionpass / fail
Formal Testing
test
generation
test suite TS
specification S
implementation i
correctness criterion
implementation
relationimp
i passes Ts
i imps
soundexhaustiv
e
![Page 11: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/11.jpg)
October 1, 2005 ARTIST2 Summer School 11
Formal Testing : Conformance
s SPECS SpecificationIUT Implementation under Test
IUT is concrete, physical object
Model the physical world
But IUT is black box ! ?
Assume that model iIUT exists
specification S
implementation IUT
correctness criterion
IUT conforms-to s
![Page 12: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/12.jpg)
October 1, 2005 ARTIST2 Summer School 12
Contents
introduction & background testing pre-orders input/output & quiescence ioco implementation relation test generation TorX test case study real-time testing
![Page 13: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/13.jpg)
October 1, 2005 ARTIST2 Summer School 13
Testing Preorders on Transition Systems
implementationi
specifications
environmente
environmente
? ? ?
i s e Env . obs ( e, i ) obs (e, s )
For all environments e all observations of an implementation i in e should be explained by
observations of the specification s in e.
![Page 14: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/14.jpg)
October 1, 2005 ARTIST2 Summer School 14
Classical Testing Preorder
LTS(L) Deadlocks(e||s)
i te s e E . obs ( e, i ) obs (e, s )
implementationi
specifications
environmente
environmente
te
![Page 15: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/15.jpg)
October 1, 2005 ARTIST2 Summer School 15
Classical Testing Preorder
implementationi
specifications
environmente
environmente
te
i te s e LTS(L) . L* . e||i deadlocks after e||s deadlocks after
FP (i) FP (s)
FP (p) = { , A | p afer refuses A }
i te s e LTS(L) . L* .
{ | e||i deadlocks after } { | e||s deadlocks after }
![Page 16: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/16.jpg)
October 1, 2005 ARTIST2 Summer School 16
Quirky Coffee Machine[Langerak]
Can we distinguish between these machines?
coin coin
tea coffeebang bang
coffee tea
coin coin
tea coffeebang bang
coffeetea
te
They are testing equivalent!
![Page 17: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/17.jpg)
October 1, 2005 ARTIST2 Summer School 17
Refusal Preorder
i rf s e E . obs ( e, i ) obs (e, s )
implementationi
specifications
environmente
environmente
rf
LTS(L{δ}) Deadlocks δ(e||
i)
e observes with δ deadlock on all
alternative actions
Deadlocks δ(e||i) =
{(L{δ})* |
e||i deadlocks after }
![Page 18: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/18.jpg)
October 1, 2005 ARTIST2 Summer School 18
Quirky Coffee MachineRevisited
coin coin
tea coffeebang bang
coffee tea
coin coin
tea coffeebang bang
coffeetea
te
rf
coin
coffee
coffee
bang
tester
δ only enabledif coffee is not
![Page 19: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/19.jpg)
October 1, 2005 ARTIST2 Summer School 19
Contents
introduction & background testing pre-orders input/output & quiescence ioco implementation relation test generation TorX test case study real-time testing
![Page 20: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/20.jpg)
October 1, 2005 ARTIST2 Summer School 20
I/O Transition Systems
testing actions are usually directed, i.e. there are inputs and outputs
L=LinLout with LinLout=
systems can always accept all inputs (input enabledness)
for all states s, for all aLin s
testers are I/O systems output (stimulus) is input for the SUT input (response) is output of the SUT
a
![Page 21: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/21.jpg)
October 1, 2005 ARTIST2 Summer School 21
Quiescence
Because of input enabledness S||T deadlocks iff T
produces no stimuli and S no responses. This is known
as quiescence
Observing quiescence leads to two implementation
relations for I/O systems I and S:
1. I iote S iff for all I/O testers T:
Deadlocks(I||T) Deadlocks(S||T)
(quiescence)
2. I iorf S iff for all I/O testers T:
Deadlocksδ(I||T) Deadlocksδ(S||T)
(repetitive quiescence)
![Page 22: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/22.jpg)
October 1, 2005 ARTIST2 Summer School 22
Input-Output QCM
coin? coin?
tea? coffee?bang? bang?
coffee!
tea?
coffee?
tea !
coffee?
tea?
tea !
states must be saturated
with input loops for
input enabledness
iote
iorf
coin!
coffee!
coffee?
bang!
coffee !
coffee?
coffee!
quiescent
states
![Page 23: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/23.jpg)
October 1, 2005 ARTIST2 Summer School 23
Contents
introduction & background testing pre-orders input/output & quiescence ioco implementation relation test generation TorX test case study real-time testing
![Page 24: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/24.jpg)
October 1, 2005 ARTIST2 Summer School 24
Implementation Relation ioco
i iorf s I/O tests T: Deadlocksδ(i||T) Deadlocksδ(s||T)
( L { } )*: out ( i after ) out ( s after
)To allow under-specification we restrict the set of traces:
i ioco s Tracesδ( s ) : out ( i after ) out ( s after
)
By adding a transition p p to every quiescent state of a systemwe treat quiescence as an observable (synchronizable) action:
δ
![Page 25: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/25.jpg)
October 1, 2005 ARTIST2 Summer School 25
i ioco s =def Tracesδ( s ) : out (i after ) out (s after
)
Implementation Relation
ioco
Correctness expressed by implementation relation ioco:
Intuition:
i ioco-conforms to s, iff
if i produces output x after trace , then s can produce x after
if i cannot produce any output after trace , then s cannot produce any output after ( quiescence )
![Page 26: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/26.jpg)
October 1, 2005 ARTIST2 Summer School 26
Implementation Relation iocoout ( i after )
=
out ( i after ?dub )
=
out ( i after ?dub.?dub )
=
out ( i after ?dub.!coffee)
=
out ( i after ?kwart )
=
out ( i after !coffee )
=
out ( i after ?dub.!tea )
=
out ( i after )
=
!coffee
?dub
?dub?kwart
?dub?kwart
i?kwart
{ }
{ !
coffee }
{ !
coffee }
{ }
{ }
{ }
i ioco s =def Straces (s) : out (i after ) out (s after )
![Page 27: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/27.jpg)
October 1, 2005 ARTIST2 Summer School 27
!coffee
?dub
?dub
?dub
i
!coffee
?dub
s
!tea
out (i after ?dub) = { !coffee } out (s after ?dub) = { !coffee, !tea }
ioco
Implementation Relation ioco
i ioco s =def Straces (s) : out (i after ) out (s after )
![Page 28: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/28.jpg)
October 1, 2005 ARTIST2 Summer School 28
!coffee
?dub
s
?dub
?dub
!coffee
?dub
i
!tea
?dub
out (i after ?dub) = { !coffee, !tea }
out (s after ?dub) = { !coffee}
ioco
Implementation Relation ioco
i ioco s =def Straces (s) : out (i after ) out (s after )
![Page 29: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/29.jpg)
October 1, 2005 ARTIST2 Summer School 29
ioco
?dub
?dub?kwart
!coffee
?kwarti
!tea !coffee
?dub
s
out (i after ?dub) = { !coffee }
out (i after ?kwart) = { !tea }
out (s after ?dub) = { !coffee }out (s after ?kwart) =
But ?kwart Tracesδ( s )
Implementation Relation ioco
i ioco s =def Straces (s) : out (i after ) out (s after )
![Page 30: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/30.jpg)
October 1, 2005 ARTIST2 Summer School 30
Contents
introduction & background testing pre-orders input/output & quiescence ioco implementation relation test generation TorX test case study real-time testing
![Page 31: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/31.jpg)
October 1, 2005 ARTIST2 Summer School 31
test
executionpass / fail
Formal Testing
test
generation
test suite TS
specification S
implementation i
correctness criterion
implementation
relationioco
i passes Ts
i iocos
?
![Page 32: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/32.jpg)
October 1, 2005 ARTIST2 Summer School 32
Test Cases
labels in L { } tree-structured finite, deterministic final states pass and fail
from each state pass, fail either one input i?
or all outputs o! and
coffee!
coin?
coin ?
tea!
coffee!tea!
coin?
pass
failfail
failpass
Test case t TTS
TTS - Test Transition System :
![Page 33: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/33.jpg)
October 1, 2005 ARTIST2 Summer School 33
Test Cases
test case t
!coin
!coin ; Start timer1
?tea fail
?timer1 fail
?coffee
!coin ; Start timer2
?tea pass
?timer2 pass
?coffee fail
coffee!
coin?
coin ?
tea!
coffee!tea!
coin?
pass
failfail
failpass
![Page 34: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/34.jpg)
October 1, 2005 ARTIST2 Summer School 34
Algorithm
To generate a test case t(S) from a transition system specification with S set of states ( initially S = {s0} )
1 end test case
PASS
Apply the following steps recursively, non-deterministically
2 supply input
supply i?
t(S after i?)
Test Generation Algorithm
3 observe output
FAIL
t(S after o!)
FAIL
allowed outputs o!
forbidden outputs
o!
![Page 35: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/35.jpg)
October 1, 2005 ARTIST2 Summer School 35
test
?-2?2
PASS PASS
otherwise
FAIL
?-3
PASS
otherwise?3
FAIL
To cope with non-deterministic behaviour, tests are not linear traces, but trees
To cope with non-deterministic behaviour, tests are not linear traces, but trees
Test Generation Example
Equation solver for y2=x
specification
? x (x >= 0)
! x
? x (x < 0)
! -x !4
!9
![Page 36: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/36.jpg)
October 1, 2005 ARTIST2 Summer School 36
Validity of Test Generation
For every test t generated with algorithm :
Soundness :t will never fail with correct implementation
i ioco s implies i passes t
Exhaustiveness :each incorrect implementation can be detectedwith a generated test t
i ioco s implies t : i fails t
![Page 37: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/37.jpg)
October 1, 2005 ARTIST2 Summer School 37
Contents
introduction & background testing pre-orders input/output & quiescence ioco implementation relation test generation TorX test case study real-time testing
![Page 38: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/38.jpg)
October 1, 2005 ARTIST2 Summer School 38
Formal Testing with Transition Systems
t: (traces){fail,pass}
traces
der : LTS (TTS)
Ts TTS
s LTS
ioco
iIUT IOTS pass
fail
obs : TTS IOTS (traces)
![Page 39: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/39.jpg)
October 1, 2005 ARTIST2 Summer School 39
Test Generation Tools for ioco TVEDA (CNET - France Telecom)
derives TTCN tests from single process SDL specification developed from practical experiences implementation relation R1 ioco
TGV (IRISA - Rennes) derives tests in TTCN from LOTOS or SDL uses test purposes to guide test derivation implementation relation: unfair extension of ioco
TestComposer Combination of TVEDA and TGV in ObjectGeode
TestGen (Stirling) Test generation for hardware validation
TorX (Côte de Resyste)
![Page 40: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/40.jpg)
October 1, 2005 ARTIST2 Summer School 40
A Test Tool : TorX
On-the-fly test generation and test execution
Implementation relation: ioco
Specification languages: LOTOS, Promela, FSP, Automata
TorX IUTobserveoutput
offerinput
nextinput
specification check
output
passfailinconclusive
user:manualautomatic
![Page 41: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/41.jpg)
October 1, 2005 ARTIST2 Summer School 41
TorX Tool Architecture
explorer
primer
driver adapter
IUTspec.
statestransitions
abstractactions
abstractactions
concrete
actions
specification
text
TorX IUTspecification
![Page 42: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/42.jpg)
October 1, 2005 ARTIST2 Summer School 42
TTCNTTCNTTCNTTCNTTCNTTCNTTCNtesttaal batch test
executionbatch test generation
TTCNTTCNTTCNTTCNTTCNTTCNTTCNtesttaal
on the fly
On-the-Fly Batch Testing
explorer
primer
driver adapter
IUTspec.
![Page 43: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/43.jpg)
October 1, 2005 ARTIST2 Summer School 43
explorer
primer
driver adapter
IUTIUTIUT bits
bytes
states
transitions
abstract
actions
transition
? x (x >= 0)
! x
? x (x < 0)
! -x
specification implementation
? x (x >= 0)
! x
? x (x < 0)
? x
On-the-Fly Testing
Concrete action
! 00001001
New menu
! x (x < 0)
! x (x >= 0)
Abstract action
! 9
Abstract action
? 3
Choice
! 9
Concrete action
? 00000011
Action
? 3
Choice
! -1
New menu
! x (x < 0)
! x (x >= 0)Check
? 3Abstract action
! -1Concrete action
! 11111111
Concrete action
? (timeout)
Abstract action
? (quiescence)
Action
? (quiescence)
Check
? (quiescence)
New menu
! x (x < 0)
! x (x >= 0)
spec
![Page 44: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/44.jpg)
October 1, 2005 ARTIST2 Summer School 44
TorX
![Page 45: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/45.jpg)
October 1, 2005 ARTIST2 Summer School 45
Contents
introduction & background testing pre-orders input/output & quiescence ioco implementation relation test generation TorX test case study real-time testing
![Page 46: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/46.jpg)
October 1, 2005 ARTIST2 Summer School 46
TorX Case Studies
Conference Protocol
EasyLink TV-VCR protocol
Cell Broadcast Centre component
Road Toll Payment Box protocol
V5.1 Access Network protocol
Easy Mail Melder
FTP Client
“Oosterschelde” storm surge barrier-control
TANGRAM: testing VLSI lithography machine
academi
c
Philips
CMG
Interpay
Lucent
CMG
academi
c
CMG
ASML
![Page 47: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/47.jpg)
October 1, 2005 ARTIST2 Summer School 47
InterpayHighway Tolling System
![Page 48: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/48.jpg)
October 1, 2005 ARTIST2 Summer School 48
Highway Tolling Protocol
Characteristics :
Simple protocol
Parallellism :
many cars at the same time
Encryption
System passed traditional
testing phase
![Page 49: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/49.jpg)
October 1, 2005 ARTIST2 Summer School 49
PaymentBox
(PB)Road SideEquipment
OnboardUnit
UDP/IPWireless
Highway Tolling System
![Page 50: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/50.jpg)
October 1, 2005 ARTIST2 Summer School 50
Test Context
ObuSim
spec
PB+
ObuSim+
TCP/IP+
UDP/IP
PaymentBox
TCP/IP
TorX
Highway Tolling: Test Architecture
PCO
SUT
UDP/IP IAP
![Page 51: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/51.jpg)
October 1, 2005 ARTIST2 Summer School 51
Highway Tolling: Results
Test results :
1 error during validation (design error)
1 error during testing (coding error)
Automated testing :
beneficial: high volume and reliability
many and long tests executed ( > 50,000 test events )
very flexible: adaptation and many configurations
Real-time : interference computation time on-the-fly testing
interference quiescence and time-outs
Step ahead in formal testing of realistic systems
![Page 52: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/52.jpg)
October 1, 2005 ARTIST2 Summer School 52
Contents
introduction & background testing pre-orders input/output & quiescence ioco implementation relation test generation TorX test case study real-time testing
![Page 53: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/53.jpg)
October 1, 2005 ARTIST2 Summer School 53
RT TorX Hacking Approaches
1. Ignore RT functionality: test pure functional behaviour
analyse timing requirements using TorX log files & assumed timing constraints
2. Add timestamps to observations adapter adds timestamps to observations when they are made and
passed on to the driver timestamps are used to analyse TorX log files
3. Add timestamps to stimuli & observations adapter add timestamps to observations when they are made and
passed on to the driver
adapter adds timestamps to stimuli when they are applied and returned to the driver
analysis:
a. timing error logging: observed errors are written to TorX log file
b. timing error failure: observed errors cause fail verdict of test case
![Page 54: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/54.jpg)
October 1, 2005 ARTIST2 Summer School 54
Real-time Testing and I/O Systems
can the notion of repetitive quiescence be combined with real-time testing?
is there a well-defined and useful conformance relation that allows sound and (relative) complete test derivation?
can the TorX test tool be adapted to support Real-timed conformance testing?
![Page 55: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/55.jpg)
October 1, 2005 ARTIST2 Summer School 55
Do We Still Need Quiescence?
coin? coin?
tea? coffee?bang? bang?
coffee!
tea?
coffee?
tea !
coffee?
tea?
tea !
Yes!
the example processes
should also be distinct
in a real-time context
coffee!
![Page 56: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/56.jpg)
October 1, 2005 ARTIST2 Summer School 56
Real-Time and Quiescence
s is quiescent iff:for no output action a and delay d: s
special transitions: s s for every quiescent system state s
testers observing quiescence take time:TestM: set of test processes having only δ(M)-actions to observe quiescence
assume that implementations are M-quiescent:for all reachable states s and s’:
if s s’ then s’ is quiescent
a(d)
δ
(M)
![Page 57: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/57.jpg)
October 1, 2005 ARTIST2 Summer School 57
Real-Time and Quiescence
i tiocoM s Tracesδ(M)( s ) :
outM ( i after ) outM ( s after
)
i tiorf s T TestM:
Deadlocksδ(i||T)
Deadlocksδ(s||T)
( L { (M) } )*:
outM ( i after ) outM
( s after )
M
![Page 58: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/58.jpg)
October 1, 2005 ARTIST2 Summer School 58
Properties
1. for all M1 M2:
i tiorf s implies i tiorf s
2. for all time-independent i,s and M1,M20
i tiorf s iff i tiorf s iff i iorf
s
M1 M2
M1 M2
![Page 59: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/59.jpg)
October 1, 2005 ARTIST2 Summer School 59
A limitation
states are saturated with input loops that reset the
clocks
x xx=ktea !
x=kcoffee!
xk
coin? coin?
tea? coffee?tea?coffee?
xk
x=kcoffee!
x=ktea !
x<Mbang?
x<Mbang?
xMbang?
xMbang?
xk xk
x x
this process cannotbe distinguished
from the next
this process cannotbe distinguished
from the previous
![Page 60: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/60.jpg)
October 1, 2005 ARTIST2 Summer School 60
Test Cases
labels in L { }, G(d) tree-structured finite, deterministic final states pass and fail
from each state pass, fail choose an input i? and a time k and
wait for the time k accepting all outputs o! and after k time unit provide input i?
or wait for time M accepting all outputs o! and
off!
x=5
x:=0
on?
x:=0
off!
x<5
off!
x=M
fail
fail
failpass
Test case t TTA
TTA – Test Timed Automata :
xM
xM
xk
x:= 0
off!
fail
![Page 61: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/61.jpg)
October 1, 2005 ARTIST2 Summer School 61
To generate a test case t(S) from a timed transition system specification with S set of states ( initially S = {s0} )
1. end test case PASS
apply the following steps recursively, non-deterministically
Timed test Generation Algorithm
allowed outputs oj! after d time-units
2. choose k (0, M) and input μ
FAIL FAIL
forbidden outputs oi! after d’ time-units
o1!
x=dn
x=d1x=d’n’
x=k
x k
tμ t1 tn
x:=0
x=d’1
on’!
μ?
o1!
on!
allowed outputs oj! after d time-units
3. wait for observing possible output
FAIL FAIL
forbidden outputs oi! after d’ time-units
x=d’1
x=dnx=d1
x=d’n’
x=M
x M
tδ t1 tn
x:=0
o1!
on’!
o1!
on!
can be calculated effectively only for
subclasses of timed transition
systems!
![Page 62: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/62.jpg)
October 1, 2005 ARTIST2 Summer School 62
Examplefail
m?
x1
x=1x:=0
x=Mx:=0
c?x=1x:=0
c?x=1x:=0
b?x=1x:=0
fail
fail
fail
pass
fail
fail
fail
failfail
x1
xM
x1
x1
xM
m? m?
t? c?b? b?
c!t?
c?t!
c?
t?
t!
c!
spec:
impl:M=k
:test
m? m?
t? c?b? b?
t!c?
c?t!
t?
t?
c!
c!
x<k
x<k
x<k
x<k
δ
c!
c!
c!
c!
c!
c!
fail
failt!
t!
t!δ
pass
t!
t!
t!x=Mδ
![Page 63: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/63.jpg)
October 1, 2005 ARTIST2 Summer School 63
Soundness & Completeness
the non-timed generation algorithm can be shown to generate only sound real-time test cases
test generation is complete
for every erroneous trace it can generate a
test that exposes it
test generation is not limit complete
because of continuous time there are uncountably many timed error traces and only countably many test are generated by repeated runs
test generation is almost limit complete
repeated test geration runs will eventually generate a test case that will expose one of the non-spurious errors of a non-conforming implementation
non-spurious errors
= errors with a
positive probability of
occurring
![Page 64: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/64.jpg)
October 1, 2005 ARTIST2 Summer School 64
Current Work
Extension of the framework M as a function of the specification state/output channel integration with symbolic data generation test action refinement robustness & tolerance in real-time testing
Extending TorX environment using CORBA IDL generate abstract TorX actions generate TTCN-3 signatures generate adapter code
Practical application TANGRAM project: testing control software for VLSI lithography
machines (ASML) smooth transition between timed & untimed testing
![Page 65: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/65.jpg)
October 1, 2005 ARTIST2 Summer School 65
Future Work
stochastic systems quality of service hybrid systems coverage measures integration white/black box spectrum ...
![Page 66: Model-Based Testing](https://reader035.vdocument.in/reader035/viewer/2022081511/56815715550346895dc4b2db/html5/thumbnails/66.jpg)
October 1, 2005 ARTIST2 Summer School 66
For more information
fmt.cs.utwente.nl/research/testing