complex test suites organization victor kuliamin [email protected] isp ras, moscow
TRANSCRIPT
Complex Test Suites Organization
Victor [email protected] RAS, Moscow
2/16
Modern Software Complexity
Modern software systems Are huge and have a
lot of functions Have very complex
interfaces Are developed by
hundreds of people
System Year Size(MLOC)Windows 3.1 1992 3Windows NT 3.1 1993 6Windows NT 3.5 1994 10Windows 95 1995 15Windows NT 4.0 1996 16.5Solaris 7.0 1998 7.5Red Hat Linux 5.2 1998 12Debian 2.0 1998 25Windows 2000 2000 29Red Hat Linux 6.2 2000 17Sun StarOffice 5.2 2000 7.6Debian 2.2 2000 59Red Hat Linux 7.1 2001 30Windows XP 2002 40Red Hat Linux 8.0 2002 50Debian 3.0 2002 105Windows Vista Beta 1 2005 50Fedora Core 4 2005 76Debian 3.1 2005 216Mac OS X 10.4 2006 86
System Year DevTeam SizeWindows NT 3.1 1993 200Windows NT 3.5 1994 300Windows NT 4.0 1996 800Debian 1.2 1996 120 *Debian 2.0 1998 400 *Windows 2000 1999 1400Debian 2.2 2000 450 *Debian 3.0 2002 1000 *Debian 3.1 2005 1500 *
3/16
Test Suites Complexity
Scalability of testing techniques is strongly related
with test suite organization
System Size(MLOC) Test suite size(MLOC)Java 1.5 5 13 (JCK)
System Year TestTeam SizeWindows NT 3.1 1993 140Windows NT 3.5 1994 230Windows NT 4.0 1996 700 (0.9)Windows 2000 1999 1700 (1.2)
System Test CasesMS Word XP 35000Oracle 10i 100000Windows XP >2000000
4/16
Traditional Test Suite Organization
Set of test cases
Suitable for test suite size management Suitable for failure analysis
− Test suites for complex systems became huge and poorly manageable
− Many common parts and low reuse− Hard to modify
Test suite
5/16
Full Set of Problems
Test execution Selection the necessary tests Test suite configuration
Test result analysis Failure diagnostics Test adequacy reporting
Test creation and maintenance Explicit relations with requirements, testing goals, and
components under test Fight with complexity Reuse
6/16
Approaches
Attributing Configuration parameters Modularization
7/16
Attributing
Adding a set of test case attributes Very simple Provides multi-aspect tests
classification Provides links to requirements and
components under tests Can help to measure test
completeness
Test suite
1 ‘x’ 3F7/
tests/usr/t1
2 ‘A’ B41/
tests/sys/t2
3 ‘x’ 9E/
tests/sys/t3
8/16
Configuration Parameters
A set of parameters that control test
execution and test results assessment Provide additional flexibility, portability and
adaptability of test suite Help to check optional requirements May be assigned dynamically− May make test suite too complex
9/16
Modularization
Separation of modules
performing specific tasks Facilitates reuse Enhances maintainability Facilitates requirements
traceability− May complicate test suite− May complicate failure analysis
10/16
“Right” Modules
Test run-time supportSchedulerTimerEvent monitor
Test adapter Test data generator Test oracle
11/16
Test Adapters
Make possible reusable and abstract tests
Test SUTAdapter
12/16
Test Data Generators
Provide data for tests Can contain filters or
selectors Can be composed from
smaller data generators
13/16
Test Oracles
Provide verdict of SUT behavior correctness
Test Data Generator
System under Test
Test OraclePartial oracles
- Event or operation oracle- Data integrity constraint
Enhance reuse Simplify requirements traceability
?
14/16
Oracle Composition
Simple caseAny failure detected by partial oracle is global Complex data integrity constraints
Behavior of distributed systemsInterleaving semantics
✕
15/16
Conclusion
Test suite organization techniques improving testing scalability already exist
The most powerful one is modularization
But usage of these techniques is restricted by existing tools
All the techniques presented need proper support in tools