Leveraging Selenium recorded tests to generate

more effective tests

Author: Cu D. Nguyen, PhDhttp://selab.fbk.eu/dnguyen/

Tuesday, January 8, 13

Selenium tests• A Selenium test consists of a sequence of

events and their input parameters.

• Example:

Event

open /CuteNews/

clickAndWait link=Add News link=Add News

type name=title afternoon news

select id=selecsinglecat label=Hot

clickAndWait css=input[type="submit"]

clickAndWait link=Logout

Tuesday, January 8, 13

Selenium tests (contd.)• Often capture/replay specific test scenarios

(or paths)

• Stick with specific input values

• The user might have to copy/paste/modify recorded tests and specify new inputs

➜ often focus on some specific functionality

➜ doesn’t scale, costly !

Tuesday, January 8, 13

We can do better with Selenium recored tests!

• How:

1. combining test paths (scenarios) to create a test machine (a map, as an analogy) to generate more paths

2. fusing available test inputs to infer input partitions, and generate new inputs from that

3. combining new paths + new inputs to create new tests

Tuesday, January 8, 13

I. Inferring test machine

Tuesday, January 8, 13

Inferring test machine from available paths

• Simple example illustrating the value of combining test paths

1. add, add2. add, remove, checkout

Available paths:

1. add, add2. add, remove, checkout3. add, checkout4. add, add, checkout

Results by combining paths:

Here we are testing a web page that has only three buttons: add, remove, and checkout

+- + Buy

want to buy! that's easy!

remove checkoutadd

Tuesday, January 8, 13

Inferring test machine (contd.)

• We can do even better: from existing test paths, we infer a test machine (or in a technical terminology: a finite state machine, FSM)

• Example:

1. add, checkout2. add, remove, checkout3. add, add, checkout

Available paths: Inferred machine:

add

add

checkout

remove

Tuesday, January 8, 13

Inferring test machine (contd.)

• From the test machine, we can generate more test paths

• Example:Inferred machine: Generated test paths

1. add, checkout2. add, remove, checkout3. add, add, checkout4. add, add, add, checkout5. add, remove, remove, checkout6. add, remove, add, remove, checkout7. …...

add

add

checkout

remove

➜ many more test paths are generated!

Tuesday, January 8, 13

2. Inferring test input partitioning

Tuesday, January 8, 13

Test input partitioning• Input space for even a single variable/parameter is

huge

• Example: price of an object can be any non negative number.

• Input space of a set of variables/parameters is even much more larger

• In software testing, we often partition input space of each input into equivalent classes

• Example: input of x is classified into 2 classes:

Tuesday, January 8, 13

Inferring test input partitions• Test input partitions can be specified by domain experts

• Or inferred automatically (when recorded/logged traces are available)

• concrete data are recored with events

• machine learning techniques can classify such data into clusters, input partitions can then be inferred from them

• Example: from all traces, at the event add(#item), the parameter #item receives the following values: {1,2,3,4,1,2,4,3,1}, we can infer that 1 ≤ #item ≤ 4. Therefore, we can partition the domain of item into 3 classes: #item < 1; 1 ≤ #item ≤ 4; #item > 4

Tuesday, January 8, 13

3. Putting them together to generate more tests

Tuesday, January 8, 13

Test generation• Test machine can generate test paths, each

path is a sequence of events

• The input of each event parameter is partitioned into classes

• Now, we can combine the two to generate more test cases

• Use pairwise criteria to reduce the number of input combination

Tuesday, January 8, 13

Example• Test sequence: add(#item), remove(#item)

• #item has 3 partitions: p1: #item < 1; p2:1 ≤ #item ≤ 4; and p3: #item > 4

• Generated new tests:

add remove

p1 p2 p3 p1 p2 p3

add(p1), remove(p1)add(p1), remove(p2)add(p1), remove(p3)add(p2), remove(p1)

……………...

Concrete values of #item can be randomly generated based on the partition information, example #item = 5 in p3

Tuesday, January 8, 13

Automated tools

Tuesday, January 8, 13

Selen2FSM & M[agi]C• Available at http://selab.fbk.eu/magic

• Selen2FSM can infer test machines (or FSMs), and input partitions from existing Selenium tests

• M[agi]C takes the FSMs and input partitions and generate executable test cases

• In Selenium Webdriver

• And Robotium for Android applications

Tuesday, January 8, 13

More info & support• Selen2FSM & M[agi]C are developed under the

support of the FITTEST project

• https://www.facebook.com/FITTESTproject

• More info

• Currently we support Selenium for web testing, and Robotium for Android application

• However, the general idea can be applied for other type of functional testing as well.

• For support, contact cunduy AT fbk DOT eu

Tuesday, January 8, 13