improving fault localization for simulink models using search-based testing and prediction models
TRANSCRIPT
.lusoftware verification & validationVVS
Improving Fault Localization for Simulink Models
using Search-Based Testing and Prediction Models
Bing Liu, Lucia, Shiva Nejati, Lionel BriandSnT Centre, University of Luxembourg
SANER 2017, Klagenfurt, Austria
Simulation
2
Simulink
• Is a data flow-driven block diagram language
• Is widely used in the automotive domain
• Is executable and enables simulation and early testing
• Supports automated code generation
3
Simulink Fault Localization
4
✔ ATestcase
Background: Statistical Debugging
5
b4
b2 b3 b5
b7b6
b8
b10 b11 b12
b14b13
b9
b1
Block Rankings
b1b2b3b4b5b6b7b8b9
b10b11b12b13b14
input1
input2
input3
output
6
Ranking Scoreb1 0b2 0b3 0b4 0b5 0b6 0b7 0b8 0b9 0
b10 0b11 0b12 0b13 0b14 0
input1
input2
input3
output
Execute test case 1b4
b2 b3 b5
b7b6
b8
b10 b11 b12
b14b13
b9
b1
Background: Statistical Debugging
7
Ranking Scoreb6 1b2 0.5b3 0.5b5 0.5b7 0.5b8 0.5b9 0.5
b10 0.5b12 0.5b13 0.5b14 0.5b1 0b4 0
b11 0
input1
input2
input3
output
Execute test case 2b4
b2 b3 b5
b7b6
b8
b10 b11 b12
b14b13
b9
b1
Background: Statistical Debugging
Problem: Performance Limitations
8
Ranking Scoreb6 1b2 0.5b3 0.5b5 0.5b7 0.5b8 0.5b9 0.5
b10 0.5b12 0.5b13 0.5b14 0.5b1 0b4 0
b11 0
• Faulty blocks may not be ranked high
• Many blocks may have the same score
• Engineers may have to inspect many blocks until they find the faulty block(s)
Goal: Improving Statistical Debugging
• Statistical debugging can be improved by using larger test suites
• But, adding test cases is not cost-free in some contexts because
* test oracles need to be developed manually
* running test cases might be expensive
• We need to generate a small but effective set of test cases9
Our Approach
10
Model Test Suite + Test Oracle Ranking
Generate a small but effective set of test cases
No
YesRegenerate rankings
Are existing rankings likely to be improved by adding more test cases?
Static analysis+
Predictor models
By diversifying test cases
Test Case Generation
• We use a Single-State search technique to generate new test cases
• Our search strategy is guided by three alternative test objectives that aim to increase the diversity of the test suite
• Dynamic basic blocks; Baudry et al. [ICSE’06]
• Coverage Density; Campos et al. [ASE’13]
• Coverage dissimilarity; Jiang et al. [ASE’09]
• None of the above test objectives require test oracles11
Stop Test Generation Criteria
• We stop the test generation when adding test cases is unlikely to improve the rankings
• We rely on
* Simulink Super Blocks: based on static analysis of Simulink models
* Predictor models built using a supervised learningtechnique and historical data
12
Decision Trees• Input features:
• Current Round index
• Set Distance
• Ordering Distance
13
Experiment Evaluation
• We applied our approach to 60 single-fault-seeded versions of our three industrial Simulink models
• We compared our three different test objectives
• We evaluated the effectiveness of predictor models
• The experiment results (except for original industry models) are available at https://github.com/Avartar/TCGenForFL/)
14
Results (1)
15
• [Accuracies of different test objectives]ü The generated test cases significantly improved the fault localization accuracy
ü The three test objectives outperform Random test generation
ü No significant difference in the three test objectives
Results (2)
16
• [Impact of adding test cases]ü Adding test cases may not always improve fault localization accuracy
Results (3)
17
• [Effectiveness of predictor model]ü Our approach is able to maintain almost the same fault localization accuracy
while reducing the average number of newly generated test cases by more than half.
Conclusion
• We proposed a new approach to improve fault localization accuracy for Simulink models by extending an existing test suite with a small number of high quality test cases.
• We evaluated our approach on 60 faulty versions of industry models.
• Our approach significantly improves the accuracy of fault localization for small test suite sizes
• Our approach is able to maintain almost the same fault localization accuracy while reducing the average number of newly generated test cases by more than half
18