test specifications a specification system for multi-platform test suite configuration, build, and...
TRANSCRIPT
![Page 1: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/1.jpg)
Test Specifications
A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution
Greg Cooksey
![Page 2: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/2.jpg)
Test Specifications-2-
Talk Outline
• Test suite challenges and basis for our new architecture
• The specification system• A walk through the new Dyninst test suite architecture
• Other features, status notes, and future plans
![Page 3: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/3.jpg)
Test Specifications-3-
Idealized Dyninst Test
• Mutator program– Uses Dyninst to analyze or modify mutatee
• Mutatee program– Simply verify that modification occurred
• Output– passed, failed, or skipped
Simple, right?
mutator
Dyninst
mutatee
outputtest
![Page 4: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/4.jpg)
Test Specifications-4-
Not That Simple• Multiple platforms
– Architecture– operating system– operating system version
– 32 vs. 64 bit
• Multiple compilers– Native– GNU– Other
• Multiple languages– C– C++– Fortran
• Multiple build parameters– Optimization levels– Stripped vs. unstripped
• Multiple test parameters– Create vs. attach
• Multiple mutator-mutatee mappings
And not all combinations are valid
![Page 5: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/5.jpg)
Test Specifications-5-
Our Goal• Write simple test cases:
• Modify specification file– Don’t modify code when adding a test
• Automatically do the rest– Build the test programs, with appropriate variations and parameters
– Run the test with appropriate parameters– Collect results and insert into database
mutator() {
insertSnippet(…)
. . .
}
foo() {
. . .
}
![Page 6: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/6.jpg)
Test Specifications-6-
New Infrastructure• Specification language and compiler
– Specifications for almost all aspects of test suite
• Test suite build system– Standard build system– Makefiles are output of spec. compiler
• Test driver– Tests are plugins– List of tests to run is output of spec. compiler
• Results parser and databaseUser only needs to write test cases and test specifications
![Page 7: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/7.jpg)
Test Specifications-7-
Specifications• What?
– Platforms• Architecture• Operating system• File name conventions• Compiler availability
– Compilers• Language• Standard flags• Option-parameter string mapping
– Languages– Test cases
• Mutator sources• Mutatee sources• Mutator-mutatee mappings• Build configurations• Run configurations
– Test suite glue• Standard libraries• Constraints
• How?– We use simple tuples, written as Prolog clauses
• Prolog lets us naturally express relationships between tuples
![Page 8: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/8.jpg)
Test Specifications-8-
Test Suite Architecture
Build Stage
Specifications
Run Stage Data Collection
Test sources
![Page 9: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/9.jpg)
Test Specifications-9-
Test Suite Architecture
Build Stage Run Stage Data Collection
spec file spec compiler
Test sources
![Page 10: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/10.jpg)
Test Specifications-10-
Specification Compiler
mutator spec
mutatee spec
test spec
standard libs
compiler spec
Prolog
Specifications
mutator tuple
mutatee tuple
test tuple
Tuples
Python
makefile
test lists
Scripts
test suite glue
![Page 11: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/11.jpg)
Test Specifications-11-
mutator(‘example’, [‘example.C’]).
mutatee(‘example’, [‘example_mutatee.c’]).
compiler_for_mutatee(‘example’, Compiler) :-
compiler_language(Compiler, ‘c’);
compiler_language(Compiler, ‘c++’).
mutator_mutatee(‘example’, ‘example’).
test_runs_everywhere(‘example’).
test_runmode(‘example’, ‘both’).
test_start_state(‘example’, ‘stopped’).
Test Specificationmutator source
mutatee source
valid compilers
mutator-mutatee mapping
build and run constraints
![Page 12: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/12.jpg)
Test Specifications-12-
Test Suite Architecture
Build Stage Run Stage Data Collection
spec file spec compiler
Test sources
![Page 13: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/13.jpg)
Test Specifications-13-
Test Suite Architecture
Run Stage Data Collection
mutator compiler
mutatee compilers
build scripts (makefiles)
spec file spec compiler
Test sources
![Page 14: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/14.jpg)
Test Specifications-14-
Build scripts
• Specification compiler generates makefile
• Mutator built as shared library plugin– example.so
• Multiple mutatees built with build parameters encoded into names– example.mutatee_g++_none– example.mutatee_g++_low– . . .
![Page 15: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/15.jpg)
Test Specifications-15-
Test Suite Architecture
Run Stage Data Collection
mutator compiler
mutatee compilers
build scripts (makefiles)
spec file spec compiler
Test sources
![Page 16: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/16.jpg)
Test Specifications-16-
Test Suite Architecture
Data Collection
mutator compiler
mutatee compilers
build scripts (makefiles)
spec file spec compilerrun scripts (test lists)
test driver src Dyninst compiler
test driver
mutator .so
mutatee1
mutatee2
mutatee3
Test sources
![Page 17: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/17.jpg)
Test Specifications-17-
Run scripts
• Prolog generates tests as tuples– (example, example.mutatee_gcc_low, createProcess, stopped)
• Python script creates lists of tests that are compiled into the test driver
• Test driver executes lists of tests and produces results output
![Page 18: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/18.jpg)
Test Specifications-18-
Test Suite Architecture
Data Collection
mutator compiler
mutatee compilers
build scripts (makefiles)
spec file spec compilerrun scripts (test lists)
test driver src Dyninst compiler
test driver
mutator .so
mutatee1
mutatee2
mutatee3
Test sources
![Page 19: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/19.jpg)
Test Specifications-19-
Test Suite Architecture
mutator compiler
mutatee compilers
build scripts (makefiles)
spec file spec compilerrun scripts (test lists)
test driver src Dyninst compiler
test driver
mutator .so
mutatee1
mutatee2
mutatee3output
logparser
database
Test sources
![Page 20: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/20.jpg)
Test Specifications-20-
Collecting Output
• Test output“example –mutatee example.mutatee_g++_low”
All tests passed
• Output is human readable• Added to database
– (example, g++, low, createProcess): passed
![Page 21: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/21.jpg)
Test Specifications-21-
Flexible Testing Structures
• Build each test as separate binary– High isolation, slow performance
• Build several tests into a single binary– Balance test isolation vs. performance
![Page 22: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/22.jpg)
Test Specifications-22-
Not Just Tests
• Specifications also used for– Platforms
•Specify architecture, operating system, filename conventions, compiler availability*
– Compilers•Specify languages*, option - command line parameter mapping, standard flags, platform availability*
– Languages•Specify filename conventions, compilers*
![Page 23: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/23.jpg)
Test Specifications-23-
Example Platform
% Linux 2.6 i386 specification
% paragraph
platform(‘i386’, ‘linux’,‘linux2.6’,
‘i386-unknown-linux2.6’).
compiler_platform(‘gcc’, ‘i386-unknown-linux2.6’).
object_suffix(‘i386-unknown-linux2.6’, ‘.o’).
library_prefix(‘i386-unknown-linux2.6’, ‘lib’).
library_suffix(‘i386-unknown-linux2.6’, ‘.so’).
![Page 24: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/24.jpg)
Test Specifications-24-
Example Compiler
% g++ specification paragraph
comp_lang(‘g++’, ‘c++’).
compiler_platform(‘g++’, Platform) :-
platform(_, OS, _, Platform), OS \= ‘windows’.
compiler_optimization_translate(‘g++’, ‘high’, ‘-O2’).
comp_std_flags(‘g++’, ‘$(CXXFLAGS)’).
comp_mutatee_flags(‘g++’, ‘$(MUTATEE_CXXFLAGS_GNU)’).
mutatee_link_options(‘g++’, ‘$(MUTATEE_LDFLAGS_GNU)’).
![Page 25: Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey](https://reader036.vdocument.in/reader036/viewer/2022062407/56649f4d5503460f94c6e0bc/html5/thumbnails/25.jpg)
Test Specifications-25-
Status
• Infrastructure functional• Converted 2/3 of old tests; working on the rest
• Will add SymtabAPI tests to new infrastructure
• Tech. report in early summer• Future: integrate into NMI Build & Test system