CS5103 Software
Engineering
Lecture 15System Testing
Testing Coverage
2
Higher level testing
Integration Testing Testing the interaction among a number of interactive
components
System Testing Testing the system as a whole, considering various
environments, external exceptions
Acceptance Testing Final user testing
Usually on GUI
3
Integration Testing
Strategies Big Bang
Top down
Bottom Up
4
Big Bang!
Prepare all relevant components
Data, Global variables…
Put them together
Pray!
5
Usage Scenario of Big Bang
Quite common in small projects
Requires no extra effort for integration
May work well if all interfaces are well-defined … Not likely to happen…
6
Top down strategy
Feature decomposition A hierarchical structure of all software features
May not require extra efforts: You should use it in requirement and design
7
An example feature decomposition
8
Top down strategy
Focus on the validation of the whole system first Test an empty system with no components at first
All components are replaced with test doubles, e.g., stubs
Gradually add components
Until all components are added
Advantages Easier to understand the process of the integration
May have a working system earlier (or always have a working system): important for modern development, such as XP
9
Top-down strategy: Illustration
10
Top down strategy
Disadvantages Requires to write test doubles
Since DOC matters in integration, so more complex test doubles is usually required
Centralized So the integration cannot be done parallelly
11
Bottom-up strategy
Focus on the integration of lowest units at first Start from an unit depending nothing else
When all sub-unit of a component are integrated, integrate the component
Until the whole system is built
12
Bottom-up strategy: Illustration
Bottom Level Subtree
Middle Level Subtree
Top Level Subtree
13
Bottom-up strategy
Advantages No test stub is required
Requires test driver but may re-use unit test cases (view the whole component as a unit)
Support parallel integration
Issues No working system is available until the end
Higher risk
Require more interactions among sub-teams working on each component
14
System testing
Test the system as a whole
Usually test against specifications
For each item in the specification Work out a test case and a test oracle
Test boundary values
Test with invalid inputs
Test with environment errors
15
Environment issues
Building Compile options and configurations
Environment variables
Third party dependencies
Underlying platforms OS, database, application server, browser
Compatibility is a huge problem Different platforms (if cross-platform)
Different versions of dependencies and platforms
Different configurations
16
Acceptance Testing
Command Line Writing test scripts
With different options and inputs
Try invalid cases : missing option parameter, invalid paths, etc.
GUI Testing
17
GUI testing
Testing Graphics User Interface is easier But harder to automate
And harder to compare results with oracles
Manual testing is still widely performed for GUI testing Manually explore the user interface
Record the steps in the test for future testings
Observe the GUI for errors
18
GUI testing
Record and Replay Screen record and replay
Record click on the screen and keyboard inputs Replay all the clicks and inputs Not robust, affected by screen size, resolution, resize,
…
Event record and replay Record all the UI events in the UI framework (swing,
MFC, android etc.), and outputs (texts) in the UI Re-trigger all the events and compare the UI texts
More robust, but requires more preparation and overhead
19
Today’s class
Higher Level Testing Software Testing Coverage
Code Coverage
Input Combination Coverage
Mutation Coverage
20
Test Coverage
After we have done some testing, how do we know the testing is enough?
The most straightforward: input coverage
# of inputs tested / # of possible inputs
Unfortunately, # of possible inputs is typically infinite
Not feasible, so we need approximations…
21
Test Coverage
Code Coverage
Specification Coverage
Model coverage
Error coverage
22
Code Coverage
Basic idea: Bugs in the code that has never been executed will
not be exposed
So the test suite is definitely not sufficient
Definition: Divide the code to elements
Calculate the proportion of elements that are executed by the test suite
23
Code Coverage
Criteria Statement (basic block) coverage, are they the same?
Branch coverage (cover all edges in a control flow graph), same with basic block coverage?
Data flow coverage
Class/Method coverage
24
Control Flow Graph
How many test cases to achieve full statement coverage?
25
Statement Coverage in Practice
Microsoft reports 80-90% statement coverage
Safely-critical software must achieve 100% statement coverage
Usually about 85% coverage, 100% for large systems is usually very hard
26
Statement Coverage: Example
27
Branch Coverage
Cover the branches in a program
A branch is consider executed when both (All) outcomes are executed
Also called multiple-condition coveage
28
Control Flow Graph
How many test cases to achieve full branch coverage?
29
Branch Coverage: Example
30
Branch Coverage: Example
An untested flow of data from an assignment to a use of the assigned value, could hide an erroneous computation
Even though we have 100% statement and branch coverage
31
Data Flow Coverage
Cover all def-use pairs in a software
Def: write to a variable
Use: read of a variable
Use u and Def d are paired
when d is the direct
precursor of u in certain
execution
32
Data Flow Coverage
Formula
Not easy to locate all use-def pairs Easy for inner-procedure (inside a method)
Very difficult for inter-procedure Consider the write to a field var in one method, and
the read to it in another method
33
Path coverage
The strongest code coverage criterion Try to cover all possible execution paths in a program
Covers all previous coverage criteria?
Usually not feasible Exponential paths in acyclic programs Infinite paths in some programs with loops
34
Path coverage
N conditions
2N paths
Many are not feasible e.g., L1L2L3L4L6
X = 0 => L1L2L3L4L5L6
X = -1 => L1L3L4L6
X = -2 => L1L3L4L5L6
35
Control Flow Graph
How many paths?How many test casesto cover?
36
Path coverage, not enough
1. main() {2. int x, y, z, w;3. read(x);4. read(y);5. if (x != 0)6. z = x + 10;7. else8. z = 1;9. if (y>0)10. w = y / z;10. else11. w = 0;12.}
Test Requirements: – 4 paths• Test Cases – (x = 1, y = 22) – (x = 0, y = 10) – (x = 1, y = -22) – (x = 1, y = -10)• We are still not exposing the fault !• Faulty if x = -10 – Structural coverage cannot reveal this error
37
Method coverage
So far, all examples are inner-method Quite useful in unit testing
It is very hard to achieve 100% statement coverage in system testing Need higher level code element
Method coverage
Similar to statements Node coverage : method coverage
Edge coverage : method invocation coverage
Path coverage : stack trace coverage
38
Method coverage
39
Code coverage: summary
Coverage of code elements and their connections
Node coverage: Class/method/statement/predicate coverage
Edge coverage: Branch/Dataflow/MethodInvok
Path coverage: Path/UseDefChain/StackTrace
40
Code coverage: limitations
Not enough Some bugs can not be revealed even with full path
coverage
Cannot reveal bugs due to missing code
41
Code coverage: practice
Though not perfect, code coverage is the most widely used technique for test evaluation
Also used for measure progress made in testing
The criteria used in practice are mainly: Method coverage
Statement coverage
Branch coverage
Loop coverage with heuristic (0, 1, many)
42
Code coverage: practice
Far from perfect The commonly used criteria are the weakest, recall
our examples
A lot of corner (they are not so corner if just not found by statement coverage) cases can never be found
100% code coverage is rarely achieved Some commercial software products released with
around 60% code coverage
Many open source software even lower than 50%
43
Input Combinatioin Coverage
Basic idea Origins from the most straightforward idea
In theory, proof of 100% correctness when achieve 100% coverage in theory
In practice, on very trivial cases
Main problems Combinations are exponential
Possible values are infinite
44
Input Combination Coverage
An example on a simple automatic sales machine Accept only 1$ bill once and all beverages are 1$
Coke, Sprite, Juice, Water
Icy or normal temperature
Want receipt or not
All combinations = 4*2*2 = 16 combinations
Try all 16 combinations will make sure the system works correctly
45
Combination Explosion
Combinations are exponential to the number of inputs
Consider an annual report system with 100 yes/no questions to generate a customized form for you (if you are not eligible for some questions, it will be not on the form)
2100 combinations = about 1030 test cases
46
Observation
When there are many inputs, usually only a small number of software inputs are related to each other
The previous example: Maybe only icy coke and sprite, but receipt is independent
A long term study from NIST (national institute of standardization technology) A combination width of 4 to 6 is enough for detecting
almost all errors
47
N-wise coverage
Coverage on N-wise combination of the possible values of all inputs
Example: 2-wise combinations (coke, icy), (sprite, icy), (water, icy), (juice, icy)
(coke, normal), (sprite, normal), …
(coke, receipt), (sprite, receipt), … (coke, no-receipt), (sprite, no-receipt), … (coke, no-receipt), (sprite, no-receipt), … (icy, receipt), (normal, receipt) (icy, no-receipt), (normal, no-receipt) 20 combinations in total
48
N-wise coverage
Note: One test case may cover multiple N-wise combinations E.g., (Coke, Icy, Receipt) covers 3 2-wise combinations
(Coke, Icy), (Coke, Receipt), (Icy, Receipt)
Note: 100% N-wise coverage will fully cover N-1-wise coverage
For K boolean inputs Full combination coverage = 2k combinations: exponential Full n-wise coverage = 4*k*(k-1)* … *(k-n+1)/n!
combinations: polynomial, for 2-wise combination, 2*k*(k-1)
49
N-wise coverage: Example
How many test cases for 100% 2-wise coverage of our example (coke, icy, receipt), covers 3 new 2-wise combinations
(sprite, icy, no-receipt), cover 3 new …
(juice, icy, receipt), covers 2 new …
(water, icy, receipt), covers 2 new …
(coke, normal, no-receipt), covers 3 new …
(sprite, normal, receipt), cover 3 new …
(juice, normal, no-receipt), covers 2 new …
(water, normal, no-receipt), covers 2 new …
8 test cases covers all 20 2-wise combinations
50
Combination Coverage in Practice
2-wise combination coverage is very widely used Pair-wise testing
All pairs testing
Mostly used in configuration testing Example: configuration of gcc
All lot of variables
Several options for each variable
For command line tools: add or remove an option
51
Input model
What happened if an input has infinite possible values Integer
Float
Character
String
Note: all these are actually finite, but the possible value set is too large, so that they are deemed as infinite
Idea: map infinite values to finite value baskets (ranges)
52
Input model
Input partition Partition the possible value set of a input to several
value ranges
Transform numeric variables (integer, float, double, character) to enumerated variables
Example: int exam_score => {less than -1}, {0, 59}, {60,69},
{70,79},
{80,89}, {90, 100}, {100+} char c => {a, z}, {A,Z}, {0,9}, {other}
53
Input model
Feature extraction For string and structure inputs Split the possible value set with a certain feature Example:
String passwd => {contains space}, {no space} It is possible to extract multiple features from one input Example:
String name => {capitalized first letter}, {not}
=> {contains space}, {not}
=> {length >10}, {2-10}, {1}, {0}
One test case may cover multiple features
54
Input combination coverage
Summary: Try to cover the combination of possible values of
inputs
Exponential combinations: N-wise coverage 2-wise coverage is most popular, all pairs testing
Infinite possible values Input partition Input feature extraction
Coverage is usually 100% once adopted It is easy to achieve, compared with code coverage Models are not easy to write
55
Specification Coverage
A type of input coverage
Covers the written formal specification in the requirement document
Example When a number smaller than 0 is fed in, the system
should report error => testcase: -1
Sometimes can be a sequence of inputs
When you input correct user name, a passwd prompt is shown, after you input the correct passwd, the user profile will be shown, …
=> testcase: xiaoyin, xxxxx, …
56
Specification Coverage
Widely used in industry
Advantages Target at the specification
No need for writing oracles
Usually can achieve 100% coverage
Disadvantages Very hard to automate
can only be automated with formal specifications
No guarantee to be complete
Quality highly depend on the specification
57
Today’s class
Unit Testing System Testing
GUI Testing
Software Testing Coverage Code Coverage
Input Combination Coverage
Mutation Coverage
58
Next class
Regression Testing Test prioritization
Total strategy
Additional strategy
Test reduction
59
Thanks!