improving software quality and devop automation with stamp, ow2con'16, paris

STAMPSoftware testing amplification to increase quality

in DevOps

1

2

DevOps

3

DevOps

Test assets are available

BUT

They are manually developed

and their quality is not sufficient

4

DevOps

The main objective of STAMP is to

automatically transform existing

test assets in order to detect

regression bugs before

production and drive down the

cost of software testing.

5

STAMP in DevOps

Continuous test amplification

6

Continuous test amplification

7

behav. diff.

Continuous test amplification

8

behav. diff.

Continuous test amplification

9

behav. diff. perf. diff.

Continuous test amplification

10

behav. diff. perf. diff.

Continuous test amplification

11

behav. diff. perf. diff.

Unit test cases amplification

12

Program

P

Input space Output space

state

Unit test cases amplification

13

Program

P

Input space Output space

state

DSpot: amplification of JUnit test suites

14

original test suite and

program under test

update in the program

under test

PDSpot

TS

P'

TS++

DSpot

•First experiments with amplification

•Automatic transformation of Junit test cases

15

P

TS

P'

TS+search-based input space amplification

observation space

amplification

run and look for differences

TS++

Search-based input space amplification

• Mutate existing test cases

• transform literal values

• replace literal values by others found in the scope

• add new invocations to methods not well covered

• create new objects

• Search-based

• ‘stack’ transformations iteratively

• select the ones that have a good fitness (e.g. cover new branches)

16

Observation space amplification

• Remove all assertions from original test cases

• keep invocations that can be inside the assertion

• Add new observation points

• run all test cases on original program

• record all values that can be observed (getters, is_*, etc.)

• Filter new observation points

• several runs of all test cases with observation points on the original program

• discard all points that vary (e.g., OS-dependent values, IO, etc.)

17

Example

18

public void testSearch() {

final ArrayStack<E> stack = makeObject();

stack.push(((E)("First Item")));

stack.push(((E)("Second Item")));

assertEquals("Top item is \'Second Item\'", 1, stack.search("Second Item"));

assertEquals("Next Item is \'First Item\'", 2, stack.search("First Item"));

assertEquals("Cannot find \'Missing Item\'", -1, stack.search("Missing Item"));}

public void testSearch_cf5() {

final ArrayStack<E> stack = makeObject();

stack.push(((E)("First Item")));

stack.push(((E)("Second Item")));

E o_5 = stack.peek();

assertFalse(((java.lang.String)o_5).isEmpty());

E o_6 = stack.pop();

assertFalse(((java.lang.String)o_6).isEmpty());

Object o_7_0 = stack.search("Second Item");

Object o_9_0 = stack.search("First Item");

assertEquals("Cannot find \'Missing Item\'", -1, stack.search("Missing Item"));

assertEquals(((java.lang.String)o_5).length(), 11);

assertEquals(((java.lang.String)o_6).length(), 11);}

new calls and asserts

extracted from asserts

new asserts

Methodology

19

Unit test ampli

Config. test ampli

Online test ampli

Methodology

20

Micro service

architecture

Unit test ampli

Config. test ampli

Online test ampli

Methodology

21

Micro service

architecture

Unit test ampli

Config. test ampli

Online test ampli

Information

managementSoftware

quality

Cloud

computing

E-Health

Smart cities

STAMP

•3-year project

•9 EU partners

•Full OSS

•Starts 01/12/16

22