improving software quality and devop automation with stamp, ow2con'16, paris
TRANSCRIPT
STAMPSoftware testing amplification to increase quality
in DevOps
1
2
DevOps
3
DevOps
Test assets are available
BUT
They are manually developed
and their quality is not sufficient
4
DevOps
The main objective of STAMP is to
automatically transform existing
test assets in order to detect
regression bugs before
production and drive down the
cost of software testing.
5
STAMP in DevOps
Continuous test amplification
6
Continuous test amplification
7
behav. diff.
Continuous test amplification
8
behav. diff.
Continuous test amplification
9
behav. diff. perf. diff.
Continuous test amplification
10
behav. diff. perf. diff.
Continuous test amplification
11
behav. diff. perf. diff.
Unit test cases amplification
12
Program
P
Input space Output space
state
Unit test cases amplification
13
Program
P
Input space Output space
state
DSpot: amplification of JUnit test suites
14
original test suite and
program under test
update in the program
under test
PDSpot
TS
P'
TS++
DSpot
•First experiments with amplification
•Automatic transformation of Junit test cases
15
P
TS
P'
TS+search-based input space amplification
observation space
amplification
run and look for differences
TS++
Search-based input space amplification
• Mutate existing test cases
• transform literal values
• replace literal values by others found in the scope
• add new invocations to methods not well covered
• create new objects
• Search-based
• ‘stack’ transformations iteratively
• select the ones that have a good fitness (e.g. cover new branches)
16
Observation space amplification
• Remove all assertions from original test cases
• keep invocations that can be inside the assertion
• Add new observation points
• run all test cases on original program
• record all values that can be observed (getters, is_*, etc.)
• Filter new observation points
• several runs of all test cases with observation points on the original program
• discard all points that vary (e.g., OS-dependent values, IO, etc.)
17
Example
18
public void testSearch() {
final ArrayStack<E> stack = makeObject();
stack.push(((E)("First Item")));
stack.push(((E)("Second Item")));
assertEquals("Top item is \'Second Item\'", 1, stack.search("Second Item"));
assertEquals("Next Item is \'First Item\'", 2, stack.search("First Item"));
assertEquals("Cannot find \'Missing Item\'", -1, stack.search("Missing Item"));}
public void testSearch_cf5() {
final ArrayStack<E> stack = makeObject();
stack.push(((E)("First Item")));
stack.push(((E)("Second Item")));
E o_5 = stack.peek();
assertFalse(((java.lang.String)o_5).isEmpty());
E o_6 = stack.pop();
assertFalse(((java.lang.String)o_6).isEmpty());
Object o_7_0 = stack.search("Second Item");
Object o_9_0 = stack.search("First Item");
assertEquals("Cannot find \'Missing Item\'", -1, stack.search("Missing Item"));
assertEquals(((java.lang.String)o_5).length(), 11);
assertEquals(((java.lang.String)o_6).length(), 11);}
new calls and asserts
extracted from asserts
new asserts
Methodology
19
Unit test ampli
Config. test ampli
Online test ampli
Methodology
20
Micro service
architecture
Unit test ampli
Config. test ampli
Online test ampli
Methodology
21
Micro service
architecture
Unit test ampli
Config. test ampli
Online test ampli
Information
managementSoftware
quality
Cloud
computing
E-Health
Smart cities
STAMP
•3-year project
•9 EU partners
•Full OSS
•Starts 01/12/16
22