Visual GUI TestingAutomated System- and Acceptance-Testing

Emil Alégroth2015-02-27

Who am I?

MSc 2010 from ChalmersPhD student since 2011Defended my Licenciate thesis 2013

And passed :P

http://www.cse.chalmers.se/~algeroth/lic.html

Layers of system abstraction

Pictorial GUI

GUI Model

System/subsystem architecture

Software components

Source code

SYSTEMInput Output

Test Automation in practice

Pictorial GUI

GUI Model

System/subsystem architecture

Software components

Source code Mutation analysis/Unit testing/etc.

Unit testing

Model-based testing/Random testing/etc.

Component/Widget based testing

?

System layer viewExample test techniques

Visual GUI Testing (VGT)

Pictorial GUI

GUI Model

System/subsystem architecture

Software components

Source code

Holistic view required!

Unit test: What line(s) of code (almost) where the defect residesbut not what feature that is broken.

VGT: What feature is defective but not what line of codeCombine different approaches: Gives you both the broken feature and potentially line(s) of code.

Regression testing

System

v1

Test suite

v1

System

v2System

v3Time

Test suite

v1Repaired

testsAdditional tests

Test suite

v1Repaired

testsAdditional testsAdditional tests

Purpose of regression testing: Verify that changes to the system has not

broken parts of the system that previously worked.

Implication: Test suite must be maintained to conform to the system’s

requirements.

Manual GUI-based RegressionTesting

Test step

Input Expected output

Result

1 Click on button x

Button x changes color

Yes

2 Click on button y

Button y changes color

.. … … …

N Click on button z

Button z changes color

Yes

Yes

NOOOOOOOOOOOOOOOO!!!!!!

No

Manual GUI-based RegressionTesting

Costly Tedious Error-prone

Regression testing in Practice

Component level (Common): XUnit - automated testing JUnit, CPPUnit, etc.

System level (Not common) Automated integration tools (Often XUnit here as

well)

GUI model (Moderately common) Selenium – Web systems QTP – Desktop

Pictorial GUI (In all companies!) Manual practices!

GUI model testing vs Pictorial GUI testing

START

STOP

GUI according to GUI model

START

STOP

Pictorial GUI shown to user

GUI model test tools: It looks great!Human: Ehhh? No!

Visual GUI Testing

Test step

Input Expected output

1 Click on button x

Button x changes color

2 Click on button y

Button y changes color

.. … …

N Click on button z

Button z changes color

Transition costly andtedious manual test-scenarios… …using image recognition

and scripts…

…to emulate end-user behaviorfor automated System and

Acceptance testing.

Require-ments

GUI test

System test

Why VGT?

Develop-ment

Commit

Build

Release

The continuous delivery cycle

Unit test

GUI TestManual:CostlyTediousError-prone

Short video demo

Benefits

Speed Test frequency

Test any GUI-based systemFlexibility

Additional Benefits

Identifies new defectsImproved quality

Easy to understand/learn Anyone/Everyone can use it

Maintenance

1 2 3 4 5 6 7 8 9 10 11

20%

7%

ContinuousVGT Maintenance

Drawbacks

Continuous assurance

Requires continuous WORK! BUT still reported as BENEFICIAL!17-62 %

WORST CASE!

Compared to other automatedtest techniques, e.g. unit tests

Recent success storyCustomer

Development company

Every 3-7 months

Manual reproductionStimulation with aVGT Script

Reproduced in < 24 hoursKilled in 7 days

What was the defect?

Custom menu

Item 1Item 2Item 3Item 4

Memory

Shows that automated tests (VGT) can identify non-frequent, non-

deterministic, defects!

VGT best practiceManual state-based test scenario

1-to-1 translation

VGT script

Possible but a bad idea!

VGT best practice

Split into scenarios

Manual state-based test scenario VGT script

Def1():

Def2():

Def3():

Def4():

Def5():

Def6():

Def7():

VGT best practiceMETA Scenarios Reused components Unique code

DefX():

Why is this best practice?

Promotes Reusability: Defs can be reused for new script

development Maintenance: Images and logic in isolated

components

Sikuli This practice is particularly important in Sikuli

since it does not support test suites!

Valid practice in all VGT tools! Also in other automated testing tools!

Practice from software development

SOFTWARE vs TESTWARE

Tools for VGT

SikuliOpen source VGT tool developed originally at

MIT

Script tool Developed in Jython Supports Python and imports of Java libraries in

its Integrated Development Environment (IDE) Has a Java Api as well Image recognition algorithm developed in C++ Scriptspråk: Python

IDE

Location(x,y)Notera: y är negativ!

Sikuli specific methods

wait(a,time)wait(time, a)

type(“c”, KEY_CTRL)type(Key.DELETE, KEY_CTRL + KEY_ALT)

type(Key.ENTER)

ENTER, TAB, CTRL, ALT, CMD, etc.

Python Syntax

NOT, AND, OR==, !=, >, <

Imports

Importing script

Imported script. In this case called: scriptA.sikuli

How do I find information about Sikuli script?

No official API, but sikuli.org has a lot of valuable information.

Instead! Search on Google “Sikuli <what you need>”, e.g. Sikuli click, Sikuli import,

etc.

For advanced script development “Python <what you need>”, e.g. Python import, Python

loops, etc.

For even more advanced scripts “JAVA <what you need>”, t.ex. JAVA I/O, JAVA jar, etc.

I have created a cheatsheet that you can use for basics: http://www.cse.chalmers.se/~algeroth/cheatsheet.html

VGT demo

Questions?Emil Alégroth: emil.alegroth@chalmers.se

www.sikuli.org

http://www.cse.chalmers.se/~algeroth/cheatsheet.html

Image settings

Example Script

More advanced settings/scripts

Sikuli OCR and Python I/O