automatic test generation from formal specifications
DESCRIPTION
Automatic Test Generation from Formal Specifications. Dr. Paul E. Black [email protected] http://hissa.nist.gov/~black/. Program under test. Specs. Specs. Specs. Program under test. test program. Stages of Automated Testing. 1 Manual 2 Custom Program Runs Tests - PowerPoint PPT PresentationTRANSCRIPT
Automatic Test Generation from Automatic Test Generation from Formal SpecificationsFormal Specifications
Dr. Paul E. [email protected]
http://hissa.nist.gov/~black/
April 19, 2023
Paul Black 2
Stages of Automated TestingStages of Automated Testing
1 Manual
2 Custom Program
Runs Tests
3 Create Program
from Spec
Specs Programundertest
Specs Programundertest
testprogram
Specs Programundertest
testprogram
TestProgram
Generator
April 19, 2023
Paul Black 3
checkmodel
Automatic Test GenerationAutomatic Test Generation
runtests
use testcriterion
.
.
.Systemspecs
Systemsource
testresults
counter-examples
CompleteTests
combinescenarios
scenarios
NISTTAO
.
.
.
http://hissa.nist.gov/~black/AFTG/
test objectives
April 19, 2023
Paul Black 4
Specification-based Coverage Specification-based Coverage MetricMetric How can we measure test set quality
without code?
checkmodel
AG(S0&a&b -> AX S1)AG(S0&(~a|~b)->AX S0)AG(S1 & a -> AX S0)AG(S1 & ~a -> AX S1)
S0 S1
a&b
a
Test case:
State 1.1: state = S0 a = 1 b = 1State 1.2: state = S1 a = 1State 1.3: state = S0
convertto CFSM
coveragemetric
=use testcriterion
.
.
.
test require-ments
# requirements satisfied total # requirements
April 19, 2023
Paul Black 5
Specification AbstractionSpecification Abstraction
How can we soundly fit high-level specification into low-level descriptions?
FiniteModeling
High LevelSystemspecsin Z, UML,VDM, Matlab,SCR, etc.
Low LevelFunctional
specs
specifications
for modelcheckers
April 19, 2023
Paul Black 6
Examples of ATGExamples of ATG Cruise Control
– 24 test cases– 87% branch coverage of code (13 of 15)– test code is 6,279 lines of Java
Reactor Safety Injection– 16 test cases, 9 after coverage minimization
Java VM Stack– 29 test cases– test code is 1,524 lines of C
Flight Guidance System– 292 test cases, 78 after coverage minimization
Secure Operating System
April 19, 2023
Paul Black 7
Current Tools (as of June 2000)Current Tools (as of June 2000)
Extract UML State Chart Mutate Test Specifications Reduce Tests Report Coverage Metric Generate Test Code