Download - 15. Regression testing
15. Regression testingTom Verheyen, Jelle Slowack, Bart Smets, Glenn Van Loon
2
Outline• Introduction
- What, why, when, how- Regression faults- Test automation- Test suite maintenance- Reducing a test suite
• Patterns- Test All- Test Risky Use Cases- Test by Profile- Test Changed Segments- Test Within Firewall
3
What is regressions testing?• Baseline version: component/system passed test suite• Delta version: component/system not passed regression
test• Delta build: executable configuration SUT containing
baseline and delta components• Regression test case: test case baseline passed, expected
to pass on delta build• Regression test suite: composition of regression test cases• Regression fault: revealed by test case that no longer
passes
4
Why should we regression test? Ariane 5
• First test flight: crash!- Data conversion from 64-bit
floating point to 16-bit signed integer
- Software (Ada) reused from Ariane 4
- No regression tests done• Estimate cost: $350 million -
$2,5 billion
5
When can we use regression test? • What kind of errors can be found?
- Delta side effects- Delta - baseline incompatibilities- Undesirable feature interactions baseline - delta- Bad fixes
• IBM: bad fixes injection rates 2% - 20%
6
When should we run a regression test?
• When a new subclass has been developed• When a superclass is changed• When a server class is changed• When a bug fix has been completed• When a new system build has been generated• When a new increment is generated for system
scope integration testing• When a system has stabilized and the final release
build has been generated
7
How to run a regression test?• Basic procedure same for all situation
1. Remove broken test cases2. Choose a regression test suite (full/reduced:
see Regression patterns)3. Set up test configuration4. Run the regression test suite5. Do something with the results
8
Regression faults• Combination of baseline B and delta D fails
- D has side effect on B- C is client/server of B- D is incompatible with B (Ariane 5!)- D’s and B’s contract specifications differ- ...
(Extended list of errors: Binder p. 761-762)• Faults at intercomponent, subsystem and system
scope
9
Test automation• Manual testing is not a good thing
- Person must reenter it and judge result- If number of tests and time to rerun
increases:1. Rerun fewer baseline test cases +
focus on tests to validate final requirements2. Add more people to enter test cases and judge results
(usually not possible)- Time to ship decrease
# regressions tests approaches 0
10
Test automation: requirements• Effective automated RT requires capabilities:
- Version control, compare baseline–delta results, smart comparator, ...
- Want to know more? Binder p.764• Test environment should be controllable
- Some variable factors can result in less-than identical test configurations• Test environment itself, generator with clock as
seed, ...- Extended list of factors: Binder p.765
11
Test suite maintenance• A regression test suite can rapidly grow large• Test decay is inevitable
- Broken testcase, redundant test cases, …
• Example: aerospace manufacturer- Test suite of 165.000 test cases
• +/- 90% test cases: redundant• Documentation inadequate• Some parts not tested: 3000 new test cases
only 18.000 test cases remained
12
Test suite maintenance: procedure1. Run baseline test suite on delta build. Remove broken test
cases.2. Correct relevant bugs.3. Merge delta component scope tests with baseline test suite.4. Use a coverage analyzer.5. Rerun the test suite (to know coverage)6. Analyze tests with same entry-exit paths (still necessary?)7. If code is not tested: develop new test case.8. Rerun test suite again.9. Check revised baseline test suite as new baseline test suit.
13
Reducing a test suite• Even with maintance, tests can become too
large- Use reduced regression test,
with selected tests
• How to “safely” reduce a regression test?- Safe: all tests that could possibly exhibit different
output when run on the delta build
14
Reducing a test suite• 4 criteria
- Inclusiveness: percentage of BL tests that may show regression faults and are in the RT suite
• Safe RT suite is 100% inclusive- Precision: percentage BL tests in a RT that cannot reveal
regression faults and are not selected in the reduced RT suite• 100 test cases: 1 cannot reach changed code
99% precise• Precise RT suite: no tests that cannot produce different
output- Efficiency: cost of identifying a reduced RT suite- Generality: range of application for the selection strategy
15
Test patterns
1. Retest All: default2. Retest Risky Use Cases3. Retest by Profile4. Retest Changed Segments5. Retest Within Firewall
16
1. Retest All• Rerun entire baseline test suite• Context
- Any scope• Fault model
- Failures because of incompatibility, side effects, undesirable feature interaction
17
1. Retest All• Strategy
- Test model: baseline test suite is reused- Test procedure: rerun after removing broken tests- Oracle: should be same results as previous run- Automation: see slides 12-13
• Entry criteria- Delta components pass component scope testing- Suitable baseline test suite exists- Test environment has same configuration as previous
runs (see slide 13)
18
1. Retest All• Exit criteria
- No pass test cases reveal bugs that are acceptable
- Remaining test cases pass• Consequences
- Inclusiveness, precision, efficiency, generality- See further slides
19
2. Retest Risky Use Cases• Use risk-based metrics to select partial RT
suite• Context
- Too few time, personnel or equipement• Fault model
- Failures because of incompatibility, side effects, undesirable feature interaction
20
2. Retest Risky Use Cases• Strategy
- Test model: risk criteria to select test cases• Suspicious use cases: unstable, complex use cases• Critical use cases: necessary for safe, effective
operation- Test procedure: Identify, develop en run test suite- Oracle: should be same results as previous run- Automation: see slides 12-13
• Entry criteria- Same as “Retest All” pattern
21
2. Retest Risky Use Cases• Exit criteria
- Same as “Retest All” pattern • Consequences
- Inclusiveness, precision, efficiency, generality- See further slides
22
3. Retest by profileIntent
- Partial regression test- Budget-constrained
Context- Too few time, personnel or equipement- Greatest dependability within budget?- Applicable at any scope
23
3. Retest by profileFault model
- Allocate tests by profile:frequency-based testing vs reliability
Strategy: Test model- # test cases for each use case? => total budget
24
3. Retest by profileExample• Total budget 6000 mins• Run 1 test case 5 mins• Chance on bug reveal 0.5 %• Bug fix requires 200 mins• Baseline test suite 20 000 test cases
How many of test cases (T) can we run?6000 = 5T + (0.005 x 200) 5=> T ≈ 5000
25
3. Retest by profileAllocation of Regression Test Time by Use Case frequency
Use case Probability Number of testsCash withdrawal 0.50 2500Deposit 0.25 1250Balance inquiry 0.25 1250
TOTAL 1.00 5000
26
3. Retest by profileStrategy: Test procedure
1. Identify, develop, run reduced test suite2. Verify: critical use cases & use case variants
included
Strategy: Oracle- should be same results as previous run
27
3. Retest by profileStrategy: Automation
- Slides 12-13- Automate selection in prev. example:
• 5 000 of 20 000 test cases=> ¼ test random selector(each test 25% chance of being selected)
Entry & exit criteria- Same as “Retest All” pattern
28
3. Retest by profileConsequences
- Inclusiveness, precision, efficiency, generality- See further slides
29
4. Retest Changed CodeIntent
- Partial regression test- Code-change analysis
Context- Too few time, personnel or equipement- Applicable at class, cluster or subsystemscope
30
4. Retest Changed CodeFault model
- See pattern “Retest All”
Strategy: Test model- Select all baseline tests that have reached:
• A changed code segment• Or a deleted code segment
-> “Graph walk technique” (Rothermel and Harrold)
31
4. Retest Changed CodeStrategy: Test model (2)
- Basic model does NOT consider:• Inheritance• Dynamic binding• Data flow• Control flow• Other dependencies
arising from state-based behaviour, iteration or recursion
32
4. Retest Changed CodeStrategy: Test procedure
1. Obtain a report from coverage analyzer, that lists codesegments by testcase
Testcase CodesegmentsT1 B1,B3,B6,B7T2 B1,B4,B8T3 B2,B5
33
4. Retest Changed CodeStrategy: Test procedure
2. Extract pairs with each segment and testcase
Testcase CodesegmentsT1 B1,B3,B6,B7T2 B1,B4,B8T3 B2,B5
PAIRB1 T1B3 T1B6 T1B7 T1..B5 T3
34
4. Retest Changed CodeStrategy: Test procedure
3. Use a version control tool to generate a report on the changes between baseline & delta
Segment ChangeB1 SameB2 DeletedB3 Changed.. ..B9 New
35
4. Retest Changed CodeStrategy: Test procedure
4. Concatenate test (step 2) & change file (step 3). Sort!
ConcatB1 SameB1 T1B1 T2B2 DeletedB2 T3...
36
4. Retest Changed CodeStrategy: Test procedure
5. Selection rules for tests:• Tests under same skip• Tests under delete include• Tests under changed include• Tests under new skip
ConcatB1 SameB1 T1B1 T2B2 DeletedB2 T3...
37
4. Retest Changed CodeStrategy: Oracle
- Same as “Retest All” patternStrategy: Automation
- see slides 12-13Entry & exit criteria
- Same as “Retest All” patternConsequences
- See further slides
38
5. Retest Within FirewallIntent
- Partial regression test- Code-dependency analysis
Context- Too few time, personnel or equipement- Applicable at class, cluster or subsystemscope
39
5. Retest Within FirewallFault model
- See pattern “Retest All”
Strategy: Test Model- Firewall = set of components whose test cases will be
included in a regression test- Firewall set is identified by analysis of changes to each
component in the SUT and its dependencies.
40
5. Retest Within FirewallStrategy: Test Model (2)• Each pair of components (A,B) is analyzed• Either or both may be changed:
1. Contract changealters external interface and/or externally visible contract(eg. alteration of public methods, ..)
2. Implementation changeother changes that are not visible to clients
41
5. Retest Within FirewallStrategy: Test Model (3)• Dependency relationship between each pair is used to
select test cases
• 4 Relationships:1. B uses A2. B is a subclass of A3. B overrides A4. B is a server of A
42
5. Retest Within FirewallStrategy: Test Model (4)• For each relationship: baseline testcase that apply to A,B
or AB may be reused in one of 4 ways:Level 0 No testcases can be rerunLevel 1 The state setup and test messages can be rerun.
Expected results must be redeveloped.The sequence of testcases may need to be
reworked.Level 2 The state setup, test messages and expected results
can be rerun.The sequence of testcases may need to be
reworked.Level 3 The test cases can be rerun as is.
43
5. Retest Within FirewallStrategy: Test Model (5)• Example
- Class Account (A) and class Money (B)- Account uses a variable amount of type Money- Implementation change to account
- Relationship: Money is a server of Account - ..
44
5. Retest Within FirewallChanges Retesting requirement
A B Dependency A ..... ... ... ... ...Implementation None B is server of A 2... ... ... ... ..
Þ Level 2 reuse of the test cases for Account
(decision table for selecting regression tests 15.7 p 791)
45
5. Retest Within FirewallStrategy: Test procedure1. Develop a dependency matrix2. Apply the decision rules (table 15.7)3. Rerun tests according to the testing levels
46
5. Retest Within FirewallStrategy: Oracle
- Same as “Retest All” patternStrategy: Automation
- see slides 12-13Entry & exit criteria
- Same as “Retest All” patternConsequences
- See next slide!
47
Pattern ComparisonInclusivenessAll: Safe, all tests selectedRisky: Unsafe, selection of testcases not by
analysis / dependenciesProfile: Unsafe, same as Risky Use CaseChanged:Safe, all baseline tests that can produce a
different result are selectedFirewall: Safe
48
Pattern ComparisonPrecisionAll: no tests are skipped, least preciseRisky: some tests that could be skippedProfile: dittoChanged: few tests that could be skipped
most precise of the white box partial regression strategies
Firewall: few tests that could be skipped
49
Pattern ComparisonEfficiencyAll: lowest analysis & setup cost
high run costRisky: time and cost are constrained (budget)
selection based on use cases (<-> implementation)
=> analysis can be done without code analyzers / technical knowledge of SUT
Profile: Same as Risky Use Casebut if the operational profile is established, then
low cost of selection analysis
50
Pattern ComparisonEfficiency (2)
Changed:High setup costRun cost = size of deltasCost can be greater than Retest All: reselection
Firewall: Highest setup costRun cost = size of firewall
51
Pattern ComparisonGeneralityAll: applicable in any circumstances
easy to set upRisky: nearly any circumstances, any scopeProfile: many circumstances,
system scope with accurate operational profile
Changed:applicable at class scoperequires skills in code analysis
Firewall: applicable at cluster scoperequires skills in code analysis