expanding our testing horizons
TRANSCRIPT
Expanding our Testing Horizons
Mark MicallefPEST Research Lab, MaltaWednesday, May 3, 2023
Two Worlds
• Understanding• Clean Solutions• Complete• Perfect
• Value• Pragmatic• Working
Idea: Catching problems earlier makes them easier to fix
Static Analysis
ParserSource Code
Analyser
IntermediateRepresentation
Analysis
Static Analysis
Source Code
The Problem Static analysis tools can identify thousands of alerts This leads to cognitive overload… … which in turn leads to… … the technique being abandoned
Concept: Actionable Alert Identification Techniques (AAIT)
Some discoveries
Priority of alert
Size/Length of the
method
Size/Length of the class
Age of alert Date of last modification
Code churn Complexity of a method
Nesting of a line of code
Code (line) coverage
0
1
2
3
4
5
Very useful Somewhat useful Slightly useful Irrelevant Missing
Num
ber
of In
terv
iew
ees
A context-specific AAIT
Expert SystemAlerts
ContextFiltered and Ranked
Alerts
Idea: We should be aware of the effectiveness of our test suites
Statement coverage can be misleading
public int multiply(int x, int y){
return (x y); }
@Test public void testMultiply(){ assertEquals(5, multiply(5, 1));}
*/
Passesx
Mutation Testing
Mutation Testing
Test Suite
Mutant 1 Mutant n
Program
Problems with Mutation Testing
Generating mutants is expensive Executing tests is expensive Susceptible to equivalent mutants Manual investigation of unkilled mutants
Applying context
Version n
δ
δ
δ
δ
Version n+1
δ
δ
Version n+1 changes
δ δ
δ
δ
δ
δ
The results
Code Churn Affected LOC
Unchanged code
Decrease in mutants
Decrease in execution time
Low 12 99.8% 91% 91%Medium 60 98.8% 62% 88%High 720 85.6% 46% 89%
Insight: Testing will never provide guarantees that bugs do not exists
Testing Activities
Test Planning Test Case Design Execution Reporting &
Management
Testing Activities
Test Planning Test Case Design Execution Reporting &
ManagementRuntime Testing
Testing(Multiple Paths) Runtime Testing
(Single user-generated Path)
Runtime Testing
Problems with Runtime Testing
Defining mathematical properties may not come naturally to people
Performance Overheads How do you fit this into your
development process?
Applying Context
Given I am a bank teller processing a transactionAnd account 67 has $100 and account 113 has $50When I transfer $50 from account 67 to account 113Then account 67 will have $50 And account 113 will have $100And this should happen within 3 secondsAnd the transfer should be loggedAnd a receipt should be printedAnd the respective clients should be notified
Relevant to feature
Cross-Cutting Criteria
Applying Context
Cross-Cutting Criteria
Feature 1 Feature 2 Feature 3
What can I do?
You can… Periodically scan academic journals for new ideas
Try Google Scholar, set up alerts for your favourite topics Sell the benefits of research to yourself and
colleagues Actively seek out collaborations with research groups,
local or otherwise Consider pursuing research for academic credit
yourself or sponsor and employee to do so
Some collaboration models
Provide a case study Sponsor an intern Hand off a problem to a research group Contribute to a research trust
